40.0 DMF (Data Management Facility) Command The B34S DMF option allows the user to maintain a data file of up to 9999 series and selectively read them into B34S. More than one DMF file can be saved inside a dataset. The DMF command can be used to: - Add series to a DMF file. - Add observations to an existing series in a DMF file. - Browse a DMF file. - Create a DMF file from current B34S data. - Convert an unformatted DMF file to a formatted file. - Convert a formatted DMF file to an unformatted file. - Delete series in a DMF file. - Read from a SCA FSV file into a DMF file - Read from one DMF file into another DMF file, - Change the frequency of data in one DMF file. - Selectively add series from a SCA FSV file to a DMF file - Rename the members of a DMF file. - Rename series in a DMF file. Note: Due to IO costs a B34S Data Bank is probably faster for really large numbers of series. The B34S Data bank, however, requires a random access file and is hard to setup and maintain. The main advantage of the DMF facility is to be able to maintain numbers of series > 99 in a B34S readable file and selectively load series for analysis. By default all DMF files are unformatted. All DMF files can be read by the B34S DATA paragraph. Except for the CONVERTUF and CONVERTFU sentences, all DMF files manipulated by the DMF command can be either formatted or unformatted. The FORMATTED DMF file is substantially slower and is intended for portability. For the same size dataset, the B34S unformatted DMF file will be smaller that the SAS *.sd1 file while the formatted DMF file will be larger. This assumes that SAS is making a default save. To avoid the possibility of data loss, sentences ADDSERIES, DELSERIES, CONVERTUF, CONTVERTFU leave the input file unchanged and create a new file on NEWUNIT. This way the old file will not be ruined should the job hang up. If UNIT=NEWUNIT for sentences ADDSERIES or DELSERIES, then the original file will be updated. Sentences BROWSE and CREATE use file INUNIT only. By default the DMF paragraph takes data from the current B34S DATA paragraph. SCA FSV files and other DMF files can be optionally read. A DMF file can be read from the B34S DATA paragraph. For further detail, see section 13.0 of the manual. A simple example is given below. Form of DMF command: B34SEXEC DMF options parameters $ ADDSERIES $ ADDOBS $ BROWSE $ CONVERTUF $ CONVERTFU $ CREATE $ DELSERIES $ FREQADJ $ RENAME $ B34SEEND $ DMF sentence parameters UNIT =n1 Sets DMF in file unit. Default = 60. The word INUNIT can be used in place of UNIT. NEWUNIT=n2 Sets DMF out file unit. Default = 61. NEWUNIT can be set the same as UNIT except for CONVERTUF and CONVERTFU sentences. The word OUTUNIT can be used in place of NEWUNIT. DMFMEMBER=k1 Sets DMF file name. Defaults to ONE. A maximum of 8 characters can be supplied. INFMT=k2 If k2= UNFORMATTED, an unformatted DMF file will be assumed. If k2=FORMATTED, a formatted DMF file will be assumed. If k2=UNKNOWN, B34S will detect the file structure. This is the default. OUTFMT=k3 If k3= UNFORMATTED, an unformatted DMF file will be assumed. This is the default. If k3= FORMATTED, a formatted DMF file will be assumed. For the RENAME sentence OUTFMT = INFMT. SCAUNIT=n3 Set SCAUNIT to read data from a SCA FSV file. If SCAUNIT is not present, the data on the current DATA paragraph will be used. SCAFNAME=k4 Sets SCA FSV file name. Default = RESIDUAL. BROWSE sentence. The BROWSE sentence allows the user to determine what series are in a DMF file. The browse sentence lists the numbers of observations, numbers of series and file comments for file allocated to UNIT. BROWSE sentence options. LISTNAMES Provides a list of series names and labels in DMF file. Example. Assume DMF file MYDATA.DMF. User wants to see what is in the file. The commands would be: B34SEXEC OPTIONS OPEN('C:MYDATA.DMF') UNIT(60)$ B34SRUN$ B34SEXEC DMF$ BROWSE$ B34SRUN$ Note: As of version 7.12g, the BROWSE sentence will automatically detect whether the DMF file is unformatted or formatted. RENAME sentence. The rename sentence allows the user to change the member names in a DMF file and or change the variable names in a DMF file. RENAME= key Set up to 8 characters for the new member name. RECODE(OLDNAME1,NEWNAME1,OLDNAME2,NEWNAME2) can be used to rename series. A maximum of 98 seriers can be recoded. Example # 1 B34SEXEC OPTIONS OPEN('C:MYDATA.DMF') UNIT(60)$ B34SRUN$ B34SEXEC OPTIONS OPEN('C:NEWDATA.DMF') UNIT(61)$ B34SRUN$ B34SEXEC DMF $ RENAME(TEST)$ B34SRUN$ will name the members TEST_____1 TEST_____2 ..... An in place rename can be done with B34SEXEC OPTIONS OPEN('C:MYDATA.DMF') UNIT(60)$ B34SRUN$ B34SEXEC DMF unit=60 newunit=60$ RENAME(TEST)$ B34SRUN$ The next two examples rename VAR1 and VAR2 as NEWVAR1 and NEWVAR2. B34SEXEC OPTIONS OPEN('C:MYDATA.DMF') UNIT(60)$ B34SRUN$ B34SEXEC OPTIONS OPEN('C:NEWDATA.DMF') UNIT(61)$ B34SRUN$ B34SEXEC DMF $ RENAME recode(var1 newvar1 var2 newvar2) $ B34SRUN$ An in place rename can be done with B34SEXEC OPTIONS OPEN('C:MYDATA.DMF') UNIT(60)$ B34SRUN$ B34SEXEC DMF unit=60 newunit=60$ RENAME recode(var1 newvar1 var2 newvar2) $ B34SRUN$ Note: In a multi dataset DMF file, the RENAME parameter can be used to pass the exact DF member name. Assume that variable XX1 in member TEST3 needs to be changed to YY. The following will perform this task in place. B34SEXEC OPTIONS OPEN('C:MYDATA.DMF') UNIT(60)$ B34SRUN$ B34SEXEC DMF unit=60 newunit=60$ RENAME(TEST3) recode(XX1 YY) $ B34SRUN$ If RENAME does not pass am argument, the first DMF file is assumed. If there are multiple XX1 names in the same dataset, only the first will be changed. FREQADJ sentence. The FREQADJ sentence allows the user to change the frequency of the data in a DMF file. FREQADJ sentence parameters. NOBSPP = n1 Number of observations per period. This is a required parameter. Data will be read from IBEGIN to IEND. If the number of observations does not divide into NOBSPP, the sample will be truncated. IBEGIN = n2 Beginning observation for new data. Default = 1. IEND = n3 Ending observation for new data. Default = last observation in new data. DMFMEMBER2=k1 Sets DMF file name for outfile. Defaults to DMFMEMBER. A maximum of 8 characters can be supplied. DISP=key If KEY = REWIND, all members on the new DMF will be replaced and the new dataset will contain only the new member. If KEY = ADD, the new member will be added to the members already on the file. ADD is the default. Example. User wants to convert monthly data to quarterly data. Both files are assumed to be unformatted. B34SEXEC OPTIONS OPEN('C:MYDATA.DMF') UNIT(60)$ B34SRUN$ B34SEXEC OPTIONS OPEN('C:NEWDATA.DMF') UNIT(61)$ B34SRUN$ B34SEXEC DMF DMFMEMBER(TEST)$ FREQADJ NOBSPP=3$ B34SRUN$ ADDSERIES sentence. The ADDSERIES sentence allows the user to add additional series to a DMF file. The new file will be on unit NEWUNIT. Unless UNIT=NEWUNIT, the old file will not be changed so as to avoid problems of ruining the original file if the job fails. The new data can be from the current B34S dataset, from an SCA FSV dataset or from another DMF file. Notes: When merging DMF files, it is important that the two DMF files be on distinct units. ADDSERIES sentence parameters. VAR(series1 series2) Specifies series to add to the DMF file. If the VAR sentence is not present, all current B34S series except the constant will be added. RENAME(OLDNAME1,NEWNAME1,OLDNAME2,NEWNAME2) can be used to rename series. RECODE can be used in place of RENAME. RENAME can be used whether VAR is used or not. A maximum of 98 seriers can be recoded. IBEGIN =n1 Beginning observation for new data. Default = 1. IEND =n2 Ending observation for new data. Default = last observation in new data. INBEGIN=n3 Sets observation at which to place series. Default=1. If n3 > 1, beginning observations will be set to missing. ADDOBS=n4 Adds observations to file. If original series do not contain data, missing observations will be added. COMMENT(' ') Adds comments to file. Comments must be LE 72 characters. MERGEUNIT=n5 Sets merge unit if the new data is in a DMF file. MERGEFMT=k1 If k1= UNFORMATTED, an unformatted DMF file will be assumed. This is the default. If k2= FORMATTED, a formatted DMF file will be assumed. MDMFMEMBER=k Sets DMF member name if merging a DMF file into a DMF file. Note: If when using the b34sexec merge command a DMF by accident contains more than one variable with the same name, the appropriate fix is to reload the data in B34S and use the b34sexec DATA sentence rename oldvar=newvar$ to rename the first occurance. Example 1. User wants to add X and Y to DMF member TEST in MYDATA.DMF. The new DMF file is NEWDATA.DMF. Series X and Y are currently active B34S variables. b34sexec options open('c:mydata.dmf') unit(60)$ b34srun$ b34sexec options open('c:newdata.dmf') unit(61)$ b34srun$ b34sexec dmf dmfmember(test)$ addseries(x y)$ b34srun$ Example 2. User wants to merge FIRST.DMF with SECOND.DMF with the result saved in NEWDATA.DMF. DMF member TEST in contained in FIRST.DMF. DMF member TWO is contained in SECOND. The merged file on unit 62 in dataset NEWDATA.DMF will contain all members that currently that are currently in FIRST.DMF. A browse is used to see what is on unit 62. b34sexec options open('c:first.dmf') unit(60)$ b34srun$ b34sexec options open('c:second.dmf') unit(61)$ b34srun$ b34sexec options open('c:newdata.dmf') unit(62)$ b34srun$ b34sexec dmf dmfmember(test) newunit=62$ addseries mergeunit=61 mdmfmember=two mergefmt=unformatted$ b34srun$ b34sexec dmf unit(62)$ browse listnames$ b34srun$ ADDOBS sentence. The ADDOBS sentence allows the user to add additional observations to an existing series in a DMF file. The new file will be on unit NEWUNIT. Unless UNIT=NEWUNIT, the old file will not be changed so as to avoid problems of ruining the original file if the job fails. The new data can come from the current B34S file or from an FSV file. ADDOBS sentence parameters. VAR(series1 series2) Specifies series for which to add observations. VAR is a required parameter. IBEGIN =n1 Beginning observation for new data. Default = 1. IEND =n2 Ending observation for new data. Default = last observation in new data. INBEGIN=n3 Sets observation at which to place series. Default= last observation in series + 1. Example. User wants to add observations to series X and Y in DMF member TEST in MYDATA.DMF b34sexec options open('c:mydata.dmf') unit(60)$ b34srun$ b34sexec options open('c:newdata.dmf') unit(61)$ b34srun$ b34sexec dmf dmfmember(test)$ addobs(x y)$ b34srun$ DELSERIES sentence. The DELSERIES sentence allows user to delete series from a DMF file. The new file will be on unit NEWUNIT. Unless UNIT=NEWUNIT, the old file will not be changed so as to avoid the problem of data loss should the job fail for any reason. DELSERIES sentence parameters. VAR(series1 series2) Specifies series to remove from DMF file. If the VAR parameter is not passed, all series are deleted. Example 1. User wants to delete X and Y from DMF member TEST in MYDATA.DMF. For safety a new file NEWDATA.DMF is created. b34sexec options open('c:mydata.dmf') unit(60)$ b34srun$ b34sexec options open('c:newdata.dmf') unit(61)$ b34srun$ b34sexec dmf dmfmember(test)$ delseries(x y)$ b34srun$ Example 2. User wants to delete DMF member TEST in MYDATA.DMF new file is NEWDATA.DMF. b34sexec options open('c:mydata.dmf') unit(60)$ b34srun$ b34sexec options open('c:newdata.dmf') unit(61)$ b34srun$ b34sexec dmf dmfmember(test)$ delseries$ b34srun$ CREATE sentence. The CREATE sentence sets up a DMF file with member name set by DMFMEMBER. The file will be placed on file allocated to UNIT. The frequency and initial year of the file can be optionally set. CREATE sentence paremeters. DISP=key If KEY = REWIND, all members on the DMF will be replaced and the new daataset will contain only the new member. If KEY = ADD, the new member will be added to the members already on the file. ADD is the default. VAR(series1 series2) Specifies series to place in DMF file. If VAR is not present all series currently in the B34S except the constant are placed in the DMF file. RENAME(OLDNAME1,NEWNAME1,OLDNAME2,NEWNAME2) can be used to rename series. RECODE can be used in place of RENAME. RENAME can be used whether VAR is used or not. A maximum of 98 seriers can be recoded. COMMENT(' ',' ') Sets comments. Comments must be LE 72 characters. HEADING(' ') Optionally sets up to 32 characters for B34S Heading IBEGIN =n1 Beginning observation for new data. Default = 1. IEND =n2 Ending observation for new data. Default = last observation in new data. INBEGIN=n3 Sets observation at which to place series. Default=1. If n3 > 1, beginning observations will be set to missing. SETFREQ(R) Sets base frequency. 1. = annual data. .1 = data once per decade. R can be set as real OR integer. If SETFREQ is passed -1, the Julian internal date is reset to unused. SETYEAR(NN) Sets base year for annual data. Frequency assumed = 1. SETMY(M1,Y1) Sets base year for monthly data. Frequency assumed = 12. SETQY(Q1,Y1) Sets base year for quarterly data. Frequency assumed = 4. SETDMY(D1,M1,Y1) Sets base year for daily data. Frequency assumed = 365. Note: The parameters SETFREQ, SETYEAR, SETMY, SETQY, and SETDMY allow the user to set the time when data in imported from an FSV file. The date will be adjusted automaticaly if INBEGIN is set to a value other than 1. The date refers to the date of the beginning observation in the DMF file. Example 1. The following commands save the current B34S data in member TEST in file MYDATA.DMF. b34sexec options open('c:\mydata.dmf') unit(60) disp=unknown$ b34srun$ b34sexec dmf dmfmember(test)$ create$ b34srun$ Example 2. The following commands variable X and Y in member TEST in file MYDATA.DMF. b34sexec options open('c:\mydata.dmf') unit(60) disp=unknown$ b34srun$ b34sexec dmf dmfmember(test)$ create var(x y) heading('x & y data') comment('data series loaded from b34s') $ b34srun$ CONVERTUF sentence. The CONVERTUF sentence converts an unformatted DMF library to a formatted DMF library. The unformatted file must be allocated to unit UNIT while the formatted unit must be allocated to unit NEWUNIT. Example b34sexec options open('c:old.dmf') unit(60)$ b34srun$ b34sexec options open('c:new.dmf') unit(61) disp=unknown form=formatted$ b34srun$ b34sexec dmf unit=60 newunit=61$ convertuf $ b34srun$ CONVERTFU sentence. The CONVERTFU sentence converts a formatted DMF library to an unformatted DMF library. The formatted file must be allocated to unit UNIT while the unformatted file must be allocated to unit NEWUNIT. Example: b34sexec options open('c:old.dmf') unit(60)$ b34srun$ b34sexec options open('c:new.dmf') unit(61) disp=unknown form=unformatted$ b34srun$ b34sexec dmf unit=60 newunit=61$ convertfu $ b34srun$ Reading DMF files from B34S DATA pragraph. The B34S DATA command will read a DMF file. For further information see section 13.7. A brief example is given below. Example 1. User wants to read series X, Y and XX from unformatted DMF file with member name TEST. The user wants to read from observation 30 to 80. The DMF file name is MYDATA.DMF. Logs of the series will be taken. b34sexec options open('c:mydata.dmf') unit=60$ b34srun$ b34sexec data filef=dmf dmfmember=test ibegin=30 iend=80$ input x y xx$ build logx logy logxx$ gen logx = log(x)$ gen logy = log(y)$ gen logxx= log(xx)$ b34seend$ Structure of DMF files. The structure of DMF files is given below. This information may be of some use of researchers who want to write external FORTRAN or C programs to construct DMF files. UNFORMATTED files are similar to FORMATTED files in that the data types are similar. Every effort has been made to simplify the structure of the DMF file to encourage the development of external programs to build DMF libraries. Header record: HEAD A2 Character*2 with value == DMFMEMBER A10 Character*10 DMF member name HEADER A32 Character*32 Optional B34S header FREQ E25.16 Real*8 Frequency of Data JULIAN I10 Integer*4 Julian Date obs 1 in DMF file NSERIES I10 Integer*4 Number of series in member NOOB I10 Integer*4 Number of observations NCOMMENTS I10 Integer*4 Number of comment cards DATE A8 Character*4 Date file was made TIME A8 Character*4 Time file was made DMISS E25.16 Real*8 Missing value for file IFTYPE I1 integer =1 for 3E25.16, =2 for 5Z16 =0 for an unformatted file. FORMATTED format (A2,A10,1X,A32,E25.16,I10/,3I10,1X,A8,1X,A8,1X,E25.16,1X,I1) UNFORMATTED format written with no blanks. Comment card format: NCOMMENT cards written A72 containing comments. Variable name / label format: NSERIES cards written: VNAME A8 Character*8 Variable name LABEL A40 Character*40 Variable label ISCHAR I1 Integer*4 =0 real, =1 character FORMATTED format A8,1X,A40,1X,I1 . Unformatted format written with no spaces. Data format DATA E24.16 Real*8 Data 4x,a8,12x Character*8 Data FORMATTED format 3E24.16 . UNFORMATTED format written with no spaces. If QUICKFSV is in effect, 5Z16 is used for format and character data can be saved in a formatted DMF file. The format used for the formatted file is NOT saved in the file so users have to set QUICKFSV as appropriate. The formatted DMF file is designed ONLY for portability. It is not intended as a production DMF file. The reason is that formatted files are substantially bigger and substantially slower than unformatted DMF files. Programming note. The OPTIONS command discussed in section 2 lists the structure of UNFORMATTED files. On the PC different compilers produce different types of unformatted files. If the users fortran or C compiler will not produce a file that is compatable with Lehey, two options are possible. One involves building the DMF file as FORMATTED and using CONVERTFU to produce the unformatted file. The other involves building the unformatted file compatible with Microsoft specifications and using the CMSTOL command available on the OPTIONS paragraph to convert the unformatted file so that it could be used with Lahey. The formatted DMF file is an excellent way to move data from one type of computer to another when there are more than 99 series. This facility similar to the SAS export facility which is illustrated next. Assume we a SAS SD2 file c:\junk\funny.sd2 and want to get means. The SAS job libname test 'c:\junk'; proc means data=test.funny; run; will read the SD2 file. The file libname misc 'c:\junk'; libname alldata xport 'c:\junk\export.dat'; proc copy in=misc out=alldata memtype=data; run; will make a SAS export file c:\junk\export.dat which can be read with the commands: * The name here is the pc or unix directory ; libname misc 'c:\junk' ; * The dataset here indicates the export file name ; libname alldata xport 'c:\junk\export.dat'; proc copy in=alldata out=misc; run; to again produce the SAS SD2 file c:\junk\funny.sd2