Jiangtang Hu | 胡江堂

In Search of Lost Time: 2016

Thu, 04 May 2017 00:00:00 +0000

2016 was gone, and I didn’t even leave one single blog post. An obvious inference should be, 2016 doesn’t exist… 2016 was a year of struggle for me as a SAS programmer. I earn my bread mostly using SAS, but is a SAS programmer like me cool? It’s not a silly question. Lots of SAS programmers feel the crunch. So, in 2016, I learned some Python, attended Pydata Carolinas, learned some Spark, and got certified by edX somehow with all the bubble words in the title, Distributed Machine Learning with Apache Spark, learned some math, Linear Algebra, learned more math, Probability, learned even more math, Numerical Analysis, and didn’t update one single blog post, because all my spare time and coffee making time were simply run out.

About

Thu, 05 May 2016 21:48:51 -0700

Jiangtang Hu.

SAS MapReduce: A Quick Followup by DS2

Fri, 04 Sep 2015 14:01:30 +0000

(DS2 would be the king!) Years ago I made up a piece of SAS code to demonstrate the basic idea of Map-Reduce. Now this idea can be best implemented by this piece of workable program with PROC DS2 (tested in SAS 9.4 TS1M2, Win7): PROC DS2; /* create some data –/ data input_data / overwrite = yes; dcl double d; method init(); dcl int i; do i = 1 to 10000000; /– create some money values –*/ d = round( (ranuni(123) * 10 ), .

A Quick Look at SAS DS2 Merge

Fri, 04 Sep 2015 12:02:39 +0000

The code: data a; input i a $ b $; datalines; 1 a1A b1 1 a1A b1 2 a2 b2 ; run; data b; input i a $ c $; datalines; 1 a1C c1 2 . c2 3 . c3 ; run; data mrge; merge a b; by i; run; proc ds2; data ds2; method run(); merge a b; by i; end; enddata; run; quit;

SAS Combinatorial Functions: Snippets

Thu, 25 Jun 2015 12:55:38 +0000

1. Permutation and Combination data null; n = 5; r = 2; *the factorial of a number; fact=fact(n); *for positive integers, fact(n) = gamma(n+1); gamm=gamma(n + 1); *C(n,r): number of combinations of n objects selected r ; *n! / [r!(n-r)!]; comb1 = comb(n,r); comb2 = fact(n) / (fact® * fact(n-r)); comb3 = gamma(n+1) / (gamma(r+1) * gamma(n-r+1)); comb4 = exp(lcomb(n,r)); comb5 = exp(lgamma(n+1)-lgamma(r+1)-lgamma(n-r+1)); *A(n,r): number of permutation (ordered arrangements);

Statistical Notes (3B): Confidence Intervals for Binomial Proportion Using SAS, Updated

Fri, 22 May 2015 22:05:39 +0000

This quick note serves as a supplementnote of my previous Statistical Notes (3): Confidence Intervals for Binomial Proportion Using SAS which I will extend as a SESUG 2015 paper. Basically I added a new Blaker method to my CI_Single_Proportion.sas file and found more CIs from SAS PROC FREQ. First of all, call the script: filename CI url “https://raw.github.com/Jiangtang/Programming-SAS/master/CI_Single_Proportion.sas” ; %include CI; %CI_Single_Proportion(r=81,n=263); and the output: The #12 is the one newly added.

Calculating Covariance by SAS, A Brutal Way

Tue, 12 May 2015 15:56:16 +0000

It was very disappointed that there is only one built-in method to calculate covariance in Base SAS: that’s in PROC CORR (while you can also do it in SAS/IML, of course): The following is a quick-and-dirty way to get a function like %COV: %macro COV(data, var1,var2); %local _cov; %let rc = %sysfunc(dosubl(%str( ods select none ; ods output cov=_cov; proc corr data=&data cov ; var &var1 &var2 ; run;

Import .Rdata to SAS, along with Labels

Tue, 12 May 2015 13:06:28 +0000

I didn’t play with SAS/IML for a while. I call it back when I need to read some R format data. Technically, .Rdata is not a data format. It’s rather a big container to hold bunch of R objects: In this example, when a .Rdata is loaded, 3 objects are included where ‘data’(the ‘real’ data) and ‘desc’ (data description portion) are of our interests. SAS/IML offers a nice interface to call R command which can be used to read the R format data:

Confidence Intervals for Binomial Proportion (Again): A Quick Note

Tue, 05 May 2015 14:22:33 +0000

In Lex’s library of the latest SAS Global Forum 2015 papers, I found an interesting paper by Wu Gong, Jeffreys Interval for One-Sample Proportion with SAS/STAT Software, where SAS MCMC procedure and a so called Random Walk Metropolis Algorithm were implemented to calculate the Jeffreys interval for binomial proportion. Years ago I wrote several posts on this topic: Statistical Notes (3): Confidence Intervals for Binomial Proportion Using SAS Statistical Notes (5): Confidence Intervals for Difference Between Independent Binomial Proportions Using SAS

List Manipulations Made Easy (and little bit of UGLY): the New DOSUBL Function

Mon, 09 Mar 2015 15:07:54 +0000

Typically, SAS list manipulations needs bunch of SAS I/O functions which are not necessarily well known to all SAS programmers. The new DOSUBL function makes this technique much more easier and little bit of ugly I must admit: %MACRO ExpandVarList(data=LAST, var=ALL); %if %upcase(%superq(data)) = LAST %then %let data = &SYSLAST; %let rc = %sysfunc(dosubl(%str( proc transpose data=&DATA(obs=0) out=ExpandVarList_temp; var &VAR; run; proc sql noprint; select name into :temp_varnames separated by ‘ ‘ from ExpandVarList_temp ; drop table ExpandVarList_temp; quit ))); &temp_varnames %MEND ExpandVarList;

Takeaway Materials From PharmaSUG SDE, Cary, NC

Mon, 29 Sep 2014 21:59:02 +0000

It’s the first PharmaSUG event I ever attended and it’s great and I plan to submit a paper for PharmaSUG 2015. Honors belonged to Mike Molter and Lex Jansen for their hardcore talks on XML and CDISC Dataset-XML respectively. Dataset-XML is basically ODM based replacement of currently wildly used SAS Version 5 Transport File. It’s an emerging technology and it’s nice to check it out: CDISC Dataset-XML Resources @CDISC

To Exit or Not Exit, the SAS Ways

Thu, 11 Sep 2014 22:55:59 +0000

The tragedy of life is not that it ends so soon, but that we wait so long to begin it. –by W. M. Lewis So, which is the exit style you prefer, in the following 3 macros (which are all valid SAS codes): /#1 if branch/ %macro tragedy_of_life1(ds); %if %sysfunc(exist(&ds)) %then %do; proc print data=&ds noobs; run; %end; %else %do; %put ERROR: Dataset &ds not exist.

Order Matters: A Weird Behavior of SAS ODS Style Options

Thu, 04 Sep 2014 14:24:16 +0000

The codes to push a dataset to Excel (technically XML): ods tagsets.excelxp file=“test.xls”; *#1; ods tagsets.excelxp options( sheet_name=’test 1′); proc print data=sashelp.class noobs; var height / style={tagattr=’format:text’} style(column)={cellwidth=.5 in}; run; *#2; ods tagsets.excelxp options( sheet_name=’test 2′); proc print data=sashelp.class noobs; var height / style(column)={cellwidth=.5 in} style={tagattr=’format:text’} ; run; ods tagsets.excelxp close; The output of #1: #2: The difference among the two pieces of code, #1 and #2 is trivial, as far as observed: only the position of the ODS style options was swapped.

Support the Blog Author to Bike MS

Mon, 11 Aug 2014 12:56:33 +0000

My dear readers, I’m asking you to support me in Bike MS (Multiple Sclerosis) and I will ride 60 miles to support the campaign in New Bern, NC, September 6 – 7, 2014. As my understanding, one of the purposes of this fundraising, is tolet the message spread. MS is a very rare disease. Although every pain is equal, we can imagine (and even understand) that few pharmaceutical companies and research institutes have strong financial incentives to support MS R&D.

A Minimal Set of Resource for SAS Programmers

Fri, 20 Jun 2014 11:04:49 +0000

A friend asked me for some recommendations of SAS books and I hope it would not be too late! If I got the similar inquiries, I will just refer to this post:) It’s not a good idea just to dump every SAS titles to beginners. If only one needed in your bookshelf, I will pick up this ultimate set: SAS Programming Course Notes (I, II, III) Typically people got such course notes by attending SAS trainings, but you can also get the electronic copies only from SAS bookstore (online of course, $100 for each).

Best Opportunity Ever For SAS Learning: Free Software, Free Online Courses

Thu, 19 Jun 2014 13:21:45 +0000

SAS Institute now offers free SAS software for non-commercial uses which contains everything you need to play with SAS programming: Base (including ODS Graphics) STAT IML SAS/ACCESS® Interface to PC Files (to deal with Excel and such) SAS Studio (which is your new online SAS editor) Note the SAS/Graph is not available but I think the newer and much fancier ODS Graphics will fill the gap. The software, SAS University Edition:

Free Clinical Trial Data (with Protocols and CRFs!)

Tue, 13 May 2014 13:55:58 +0000

Bookmark it, a collection of clinical data, with full protocols and annotated CRFs by National Institute on Drug Abuse (NIDA): http://datashare.nida.nih.gov/protocol/data

Use List Object in SAS: Yet Another Undocumented Feature in SAS 9.4

Tue, 01 Apr 2014 23:42:24 +0000

Last year I gave a talk in SESUG 2013 on list manipulation on SAS using a collection of function-like macros. Today I just explored in my recently upgraded SAS 9.4 that I can play with list natively, which means I can create a list, slice a list and do other list operations in Data Steps! This is not documented yet(which means it will not be supported by the software vendor) and I can see warning message in Log window like “WARNING: List object is preproduction in this release”, and it is still limited somehow, so use it in your own risk (and of course, fun).

Record Qualifier v.s. Variable Qualifier

Tue, 18 Mar 2014 11:00:36 +0000

If you think it’s clear enough to differentiate record qualifier from variable qualifier as they were defined (in CDISC SDTM Version 1.3): Record Qualifiers define additional attributes of the observation record as a whole (rather than describing a particular variable within a record). Examples include –REASND, AESLIFE, and all other SAE flag variables in the AE domain; AGE, SEX, and RACE in the DM domain; and –BLFL, –POS, –LOC, –SPEC, and –NAM in a Findings domain

A SAS Note for Length Limit of Strings in CDISC Datasets

Tue, 18 Feb 2014 23:37:22 +0000

Clinical programmers are very familiar with the length limit of strings in CDISC compliant datasets, such as #1: variable names: <= 8 characters #2: variable labels: <= 40 characters #3: data set labels: <= 40 characters #4: data value of a single variable: <= 200 characters and they are due to the limitations of SAS XPORT transport files (v5). But there are some rules which are not as explicit as rule #1-4, for example,

SAS Format Viewer and Others (EG)

Thu, 06 Feb 2014 23:28:26 +0000

Today I tried to install a SAS format viewer, one of my favorite SAS AF utilities in a virtual machine, and it came out a bad news and a good news. The bad news is that the website I used to download the SAS Format Viewer (by Frank Poppe) is no longer accessible(LESSON LEARNED: make an offline copy when you check piece of codes online!). While the good news is, that a company named BiX just released a free version of (yet another) SAS format viewer.

SAS Proc Groovy in Action: JSON File Processing

Thu, 12 Dec 2013 13:30:49 +0000

Last year I took a bite of the newly SAS Proc Groovy to read JSON data since there was no direct “proc import” or “infile” or “libname” way to play with JSON. Here is an nice example from SAS official blog, by Falko Schulz where Proc Groovy is used to parse Twitter JSON file: How to import Twitter tweets in SAS DATA Step using OAuth 2 authentication style PS: Since SAS 9.

The Hardware and Software 2013 I’m Most Thankful For

Fri, 29 Nov 2013 00:16:33 +0000

It’s time of year to give thanks. As a programmer, e-book reader, blog writer and web surfer, I should express my sincere appreciation to such hardware and software (I use majority of them at daily basis and most of them are free): 0. Hardware Lenovo Thinkpad W520 (This is not free): my workhorse machine, now replaced by W530. 1. Google Stuff Google Chrome (Windows and Android apps): The first thing to do when getting a new machine is using IE to download the Chrome, then I can just keep moving.

SAS Data Driven Programming: My 4 Favorite Techniques

Thu, 07 Nov 2013 10:10:21 +0000

I use relatively fixed patterns in my SAS programming life. For so called data driven programming (or dynamic programming), I used the following 4 techniques, chronologically: macro array call execute list processing for each loop For a quick demo, I will start with a simple scenario in which the data set sashelp.zipcode should be spitted to pieces of datasets by states (in real projects, the codes would be more complicated but share the simple atom structure).

3 Ways to Convert SAS Datasets to Plain Codes

Mon, 04 Nov 2013 21:19:49 +0000

1. If you have SAS Enterprise Guide installed Try one of Chris Hemedinger’s EG plug-ins, “_Data Set to DATA Step_”. Chris also wrote a post for it, Turn your data set into a DATA step program. 2. If you need to embed the dataset to SQL Clauses Use one of Eric Gebhart’s ODS tagsets, “SQL Tagset”. 3. If you want to make your hands dirty… You can also take a bite of some SAS base codes by Rick Langston.

Community Contributors to SAS/STAT

Wed, 18 Sep 2013 19:24:42 +0000

I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail. – Abraham Maslow, The Psychology of Science, 1966 Yes it’s true: since I have a pretty rich collection of SAS list processing macros, I’d like my list container to hold everything itemized. Here is a nail I found in SAS/STAT online help. There are strong community contributing to the development of SAS/STAT software.

SAS ODS Tagsets Guru Eric Gebhart on Github

Tue, 17 Sep 2013 15:59:12 +0000

Check it out: https://github.com/EricGebhart Also, his homepage: http://ericgebhart.com/ Shameless Ad: I will have a talk on Github for SAS programmers in the forthcoming SESUG (Oct 20-23, 2013) at St.Pete Beach, FL. Welcome to drop by if you are on the spot!

How I Use SAS?

Mon, 16 Sep 2013 12:48:31 +0000

It’s pretty fun to read SAS/Graph expert Robert Allison’s note, Using SAS- A Case Study on his SAS programming environment, such as SAS version, OS, fileserver and editors. Here is my follow up: SAS Version I always want to keep pace with the latest SAS software; something I got it, sometime not. As of September 2013, I use SAS 9.4 in my workstation. For different tasks, clients or their combinations, I should use specific versions of SAS in customers’ computing environment remotely or in virtual machines; they are mostly SAS 9.

SAS System Viewer Doesn’t support SAS 9.4 Datasets

Tue, 27 Aug 2013 09:42:36 +0000

This is sad, really sad. SAS System Viewer is not officially supported by SAS since the launch of SAS Universal Viewer, but it can still open SAS 9.3 datasets and I love it. SAS System Viewer is mush superior than SAS Universal Viewer and any other data viewers in Enterprise Guide, JMP, SAS Drug Development and SAS Data Integration Studio regarding response time. [update 2014/03/18] A new SAS 9.4 data set option “EXTENDOBSCOUNTER” prevents SV to read the data set correctly, see

Noninferiority Testing with SAS

Mon, 26 Aug 2013 22:59:27 +0000

I planed to extend nonWinferiority testing to one of my statistical notes, Equivalence Testing and TOST (Two One-Sided Test) since noninferiority test is simply half part of the equivalence test. Today I’m glad to find an even better explanation from SAS Usage Note 48616: Design and analysis of noninferiority studies You will love it if you are interested in this subject (bookmark this link since it looks like unfinished somehow).

Confidence Intervals for Difference Between Independent Binomial Proportions: A SAS 9.4/STAT 12.3 Update

Sun, 25 Aug 2013 23:32:29 +0000

For the computing of confidence intervals for difference between two independent binomial proportions in SAS 9.4/STAT 12.3, it’s hard to figure out exactly how many methods are available because there is no “CL=ALL” trigger in RISKDIFF option (but you can get one in BIONOMIAL option to compute the confidence intervals for binomial proportion). As far as I hacked, I got 11 intervals (last year I tried SAS 9.3 and got 8; codes available below):

Everything You Need to Know about Your SAS Session and No More…

Sun, 25 Aug 2013 23:20:46 +0000

You will get rich information for your SAS session(definitely more than you need) by submitting: %put all; *all system macro variables; proc setinit; *all licensed software; run; proc product_status; run; proc options; *options; run; proc javainfo; *Java infor; run; proc registry list; *registry; run;

Confidence Intervals for Binomial Proportion: A SAS 9.4/STAT 12.3 Update

Thu, 22 Aug 2013 10:57:01 +0000

1. A new confidence limits method added, Wilson (Corrected) 2. The binomialc option is no longer documented, but still works After getting the access to SAS 9.4 alone with SAS/STAT 12.3, I first took a look at my favorite SAS statistical procedure, PROF FREQ. Last year I had a note on confidence intervals for binomial proportion based on SAS 9.

Yet Another Instance of DoW-Loop

Wed, 07 Aug 2013 23:49:55 +0000

Just got another DoW-Loop example from SAS Sample #48582: http://support.sas.com/kb/48/582.html

Good Morning, Hotfix Tool for SAS 9.4

Mon, 29 Jul 2013 09:56:11 +0000

The first news I got from today’s inbox is that SAS Hot Fix Analysis Download and Deployment Tool (SASHFADD) for SAS 9.4 is released today. Just yesterday I found it was not available. I did a quick launching and it works fine. Check it out: http://ftp.sas.com/techsup/download/hotfix/HF2/SASHFADD.html

Open Box: SAS 9.4 in Windows 7

Sun, 28 Jul 2013 16:06:37 +0000

That’s a record in my personal SAS software adoption: SAS 9.4 was released at 10 July 2013 and I just got it installed two weeks later! Here my first-day notes with SAS 9.4: 1. SAS Foundation 9.4 can be co-existed with SAS 9.3. If SAS 9.3 was already installed, the default of path of 9.4 is C:\Program Files\SASHome2; otherwise, it will take over C:\Program Files\SASHome. 2. But for SAS PC Files Server 9.

List Processing With SAS (2): List Creating II

Fri, 26 Jul 2013 00:24:50 +0000

%range is a genetic list creator. To apply data driven programming technique, we need to fetch metadata from source data dynamically, for example, to get all variables from a input dataset. You can easily make it by 1. PROC SQL, from a SAS dictionary table, or macro variable by SELECT INTO; or 2. Proc Contents; or 3. even a smart data step with RESOLVE function and with CALL SYMPUT, like

The Return of SAS System Viewer

Tue, 16 Jul 2013 22:37:50 +0000

SAS Universal Viewer (UV) jumped out since SAS 9.2 and in most cases, it’s installed as replacement for the SAS System Viewer(SV) but I reinstalled SV in all machines where I had the right privileges. UV has some nice new feature but for me it is extremely and unacceptably slow. Every time I open a relatively not entirely small dataset (literally, 30MB) in UV, it shows up “initializing” forever (even the initializing process is done, I guess).

List Processing With SAS (1): List Creating I

Thu, 11 Jul 2013 21:10:14 +0000

Suppose you have 10 datasets, literally ds1, ds2, …ds10 and you need to concatenate them all. You may first get a quick shortcut set ds1-ds10. If such list members were generated dynamically (and may hold a form like ds1a, ds2a,… ds10a) , you will probably come out a macro solution: %macro doit; data combine; set %do i=1 %to &n; ds&i %end; ; run; %mend; %doit

Catch Up with SAS

Thu, 11 Jul 2013 00:10:28 +0000

SAS 9.3 was released at July 2011 when I former company just successfully launched an upgrading to SAS 9.2. SAS 9.2 came to life at March 2008 (it’s a big leap) and at that time I played with SAS 9.1.3 heavily. Yesterday(10Jul2013) was the official release day for SAS 9.4 while I just got my SAS 9.3 installed at this March! Based on historical data, I will probably play with SAS 9.

Free Utilities to Zip Files in Windows: Quick Notes

Sat, 29 Jun 2013 10:36:59 +0000

1. Powershell cmdlet Write-Zip from PSCX Pro: flexible, easy to use Con: relatively time consuming 2. 7-Zip Command-Line Pro: flexible, relatively quicker, easy to be embedded in other scripts Con: (…) 3. Jar command in JDK Pro: flexible, relatively quicker, easy to be embedded in other scripts Con: (…)

Preproduction in SAS

Sat, 29 Jun 2013 09:51:52 +0000

It seems the list of SAS preproduction is becoming shorter. Thanks for any inputs (just tested in SAS 9.3): 1. Output Data Step View (v.s Input Data Step View) WARNING: The definition of an output DATA step view is an experimental feature in this release and is not intended for use in the development of production applications. 2. ODS Report Writing Interface (will turn to production in SAS 9.

List Processing With SAS: A Github Repository

Sun, 31 Mar 2013 22:24:02 +0000

I have a function like macro (recursive version) to create a sequence: %macro _list(n,pre=ff); %if &n=1 %then &pre.1; %else %_list(%eval(&n-1)),&pre.&n; %mend _list; %put %_list(3); *produces ff1, ff2, ff3; But when I read one of Ian Whitlock’s papers, Names, Names, Names – Make Me a List (SGF 2007, SESUG 2008), I say: stop! I’m gonna use Ian’s %range and I create Github page to hold it (with minimum modifications due to personal preference).

Localize Your Macro Variable? Mostly Not Needed or Do It If You Only Want to Initiate It

Fri, 29 Mar 2013 19:22:46 +0000

[NOTE2013-05-30: This post is permanently suspended. In this post, I only focused on the initialization effect of the %local statement and totally ignored its function to avoid variable collisions as Quentin mentioned in comment box.] A piece of SAS codes to create a list of all variables within a dataset (a nice programming trick from Art Carpenter, 2004): %macro getvars(dset) ; %local varlist ; %let fid = %sysfunc(open(&dset)) ;

What’s New

Wed, 20 Mar 2013 23:09:08 +0000

I didn’t blog for a while in this first half March and there are bunches of new stuff to catch up: I had a new baby! He was delivered on time (and on budget!), lions tigers and bears, oh my… His brother is Tiger so I named him, Leo. And I got the latest SAS 9.3 (TS1M2) installed! SAS is jus getting much beautiful. OpenCDISC had the latest release, Version 1.

New Game in Town: SAS Metadata Administration

Wed, 13 Mar 2013 21:46:35 +0000

You might also read that there were constant (although still not frequent) posts on SAS metadata querying and other administration tasks in SAS blogosphere since 2012 (when I started to play with it^). It is yet another evidence that more and more SAS programmers switched from SAS foundation to the so called SAS intelligent platform. In the traditional SAS foundation world, a nice source of metadata is the Dictionary table (or V datasets in SASHELP library); In SAS intelligent platform, a much more comprehensive metadata is stored in a SAS Metadata Server (Is “Data” singular or plural?

SAS Snippet: Reshape Data Using SAS DoW Loop (From Long to Wide)

Mon, 11 Mar 2013 21:10:28 +0000

In Art Carpenter’s latest book, Carpenter’s Guide to Innovative SAS Techniques, a data step approach to transpose data (from long to wide) works like (Ch2.4.2): data tst; input type $ grp value $3.; datalines; A 1 a A 2 aa A 3 aaa B 1 b B 2 bb B 3 bbb C 1 c C 2 cc C 3 ccc ; data art(keep=type grp1-grp3); set tst; by type; retain grp1-grp3 ; array grps {3} $ grp1-grp3; if first.

Yet Another Undocumented Feature (new in SAS 9.3)

Tue, 26 Feb 2013 21:00:14 +0000

Today just caught an undocumented SAS feature. I ran the following SAS codes to delete a dataset(in Windows 7 with SAS 9.3): data a; b=1; run; proc datasets noprint; delete a; run; quit; and the dataset was deleted as I expected: 32 proc datasets noprint; 33 delete a; 34 run; NOTE: Deleting WORK.A (memtype=DATA). 35 quit; But when tested in a SAS 9.2 machine, I got errors

How to Write a Check? Use SAS Format!

Sun, 24 Feb 2013 21:59:02 +0000

During my initial stay in US last year, one of the interesting exercises was to write a check. The Arabic numerals (the universal language!) were pretty intuitive while I didn’t feel much comfortable on spelling the face value in words: I never played the game! But the good side of this story was, as a SAS programmer and I used a WORDFw. Format: data null; money=8.

Github for Clinical/Statistical Programmers

Wed, 20 Feb 2013 22:01:03 +0000

PhUSE-FDA Working Group 5 (Development of Standard Scripts for Analysis and Programming) just adopted Google Code as collaborative programming platform. Google Code is one of the most popular and respected open source software hosting sites in the world and it is definitely a good choice for PhUSE-FDA WG5. But after viewing one of WG5’s working reports, Sharing Standard Statistical Scripts and getting to know why they finally chose Google Code (rather than Github which was also tested by WG5 members), I think it’s necessary to clarify some misunderstanding against Github where I’m also an occasional user.

Linguistic Sorting in SAS Proc Sort

Tue, 19 Feb 2013 23:31:04 +0000

Just took a look at the linguistic sorting features in SAS Sort procedure, and got some neat options to apply to my task. For example, I want to sort ID in the following dataset: data t1; input ID $ ; datalines; T20 T4 T3 T1 ; and want to get such intuitive orderings (files sorting in Window 7 directory): But when apply the default sorting: proc sort data=t1 out=t2; by ID; run;

SAS ODS Report Writing Interface: A Quick Demo

Sun, 17 Feb 2013 20:21:00 +0000

I personally nominated SAS ODS Report Writing Interface was the one of the best technology I found in SAS in 2012. It can generates reports cell by cell and row by row and has much flexibility to produce highly customized reports. Basically, to use it, first assign an ODS destination. Nothing new, and I prefer HTML like ods html file=“output.html” style=sasweb; then create a object(instance) based on the ODS Report Writing class, odsout, declare odsout myout();

LearningR!

Fri, 08 Feb 2013 23:37:49 +0000

I spent almost all of my blogging time last month to follow an online course in Coursera, Computing for Data Analysis (with R) by Dr. Roger Peng of Johns Hopkins, Biostatistics Department. I already checked out bunch of Coursera courses just to take a look at what else MOOC look like, this R course was the first and only one I took seriously: I reviewed all slides (although didn’t watch the videos since I didn’t have enough time) and most important, finished all programming assignments(and got full score!

How to Jump into SAS Data Integration Studio

Sat, 12 Jan 2013 18:08:14 +0000

This is for for SAS programmers who might be reluctant to check out a GUI tool like SAS Data Integration Studio (DIS for short, which is graphic tool to implement ETL processes: extract, transform, load). DIS translates all users dragged nodes, transformations and process into SAS codes which are traditionally written by SAS programmers. I find one of the benefits of using DIS is that I can package all the ETL work to other players(technical or nontechnical) then they can replay the job by dragging and clicking.

How to Get Row Numbers in SAS Proc SQL (and DO NOT Use the Undocumented MONOTONIC Function)

Fri, 11 Jan 2013 15:01:09 +0000

SAS programmers are longing for row number function used in Proc SQL, like ROW_NUMBER() in Oracle SQL and it will act like data step system variable N. When you google this question, most likely you will get MONOTONIC() function, which might be one of the most famous undocumented features shipped by SAS. You can of course use it, but at your own risk! In SAS Usage Note 15138, it’s said:

Get Started with SAS Tagsets.ExcelXP

Sat, 05 Jan 2013 17:46:24 +0000

Hate it or not (Yes I do), SAS programmers can’t just get rid of Microsoft Office Excel in their life. Now my turn (with Tagsets.ExcelXP, I can at least get rid of DDE)… 0. SAS Templates window A visual way to browse all ODS templates is to use windows command “ODSTEMPLATES” to invoke a SAS template window. 1. Check the current ExcelXP tagset version filename temp temp; ods tagsets.

Blog Statistics: 2012

Tue, 01 Jan 2013 23:47:44 +0000

(This blog’s page view data from Google Analytics; 43 posts in total in 2012) Top 10 countries/territories and cities Most of my readers come from US and I’m glad my Chinese fellows still fork me since I moved to US this year (today is my one year anniversary in US). To my surprising there are much more readers from South Korea than from Japan(#12) and Singapore(#22). Readers from a secret city (#7) are watching me!

Best of SAS: A Personal Nomination 2012

Mon, 31 Dec 2012 01:13:03 +0000

There SAS applications/procedures/features were not necessarily available since 2012. This year I paid special attention to them when began to use SAS 9.3. The following notes are totally my personal endorsement purely based my own experience as a user: XML File Reading: SAS XML Mapper SAS XML Mapper itself is not an elegant tool from software design perspective (an example: I keep multiple versions because the latest version seems not carry out all the functionalities from old ones), but it is best XML file processing tool for SAS programmers like me.

The Great, Open, Vendor-neutral, Platform-independent Data Standards, . . . Yet in PDF Formats

Sat, 29 Dec 2012 17:47:36 +0000

You know I mean the CDISC standards including CDASH, SDTM, SEND, ADaM, … and you are right there are few not only in PDF format (ODM, define.xml for example). Today Jozef (Jos) Aerts from XML4Pharma posted his frustration of copying and pasting metadata from only PDF-formed SDTM-IG 3.1.4. I hate to complain the volunteer work by the CDISC team but it is worth a discussion, is there any better way to publish CDISC Standards?

Weekend Clips: Data Scientist Episode II

Sun, 23 Dec 2012 15:14:13 +0000

1. What is A Data Scientist Anyway? 2. You Just Can’t Be Replaced by Yourselves! 3. We are all Data Scientists!

SAS Certified, Again

Sat, 22 Dec 2012 23:13:39 +0000

Last week I got another SAS certification, SAS Certified Data Integration Developer for SAS 9 and I must say it can’t be better as a SAS holiday gift for myself! I took all my SAS certificates when employed (when in Sanofi Pasteur, 2010, I got two SAS programming certificates, Base and Advanced). You may say it’s kind of certificate for sake of certificate and not for applying a job like college students.

My SAS Books: Shopping List 2012

Fri, 07 Dec 2012 23:36:30 +0000

Last year I threw away all my SAS books (to friends and colleagues in Beijing) before moving to US. You might agree that it’s not economically bound to transport such heavy books intercontinentally! Now I just start to build my SAS library one by one. I will most probably not buy more SAS books as Christmas gifts for myself, so it’s time to take a 2012 snapshot of shopping:

Weekend Clip: Data Scientist

Fri, 30 Nov 2012 22:31:52 +0000

Two tweets: One blog post: _Statisticians aren’t the problem for data science. The real problem is too many posers_ One job advertisement: SAS Data Scientist ?(!) A joke: The biggest joke about data scientist is that the Google query “data scientist joke” returns nothing interesting.

Extract the Version of SAS and OS of a SAS Format or Macro Catalog: A Little Bit of Perl Regular Expression

Sun, 25 Nov 2012 20:36:52 +0000

SAS Sample 34444(Determine the operating system in which a format catalog was created) posts piece of codes to get the version of SAS and OS of a SAS format catalog. It is useful since a SAS catalog can be only read in the operating systems same to its source machine. 2 cents add to this note: First, we do not need to write codes to get to know the machine version for a catalog(formats.

WordPress and Black Friday

Fri, 23 Nov 2012 23:04:03 +0000

Yes it’s BLACK that today this WordPress based blog was down for a while with a 500 error: The website encountered an error while retrieving http://www.jiangtanghu.com/blog/. It may be down for maintenance or configured incorrectly. I checked web log and it says: [24-Nov-2012 02:58:49] PHP Parse error: syntax error, unexpected T_STRING, expecting ‘,’ or ‘;’ in ../public_html/jiangtanghu/blog/wp-content/plugins/wp-conditional-captcha/wp-conditional-captcha.php on line 401 So the problem comes from a WordPress plugin “Conditional CAPTCHA for WordPress” .

Macro Quoting in SAS Data Integration Studio

Mon, 19 Nov 2012 21:35:04 +0000

You can run the following piece of codes successfully in these 3 SAS programming environments: BASE SAS Enterprise Guide: create a new “File-New-Program” SAS Data Integration Studio: create a new “Tool-Code Editor” %let species=“Setosa” “Versicolor”; data a; set sashelp.iris; where species in (&species); run; Then create a Transformation in SAS Data Integration Studio (DIS for short; I use version 4.

A SAS 9.3 Macro Trick: %put &=var

Thu, 08 Nov 2012 20:09:50 +0000

This is new since SAS 9.3 on how to display macro variable name and its value. Try to run %let var=1,2,3; %put &=var; or %macro test(var); %put &=var; %mend; %test(%str(1,2,3)) and you will get in Log window VAR=1,2,3 You can read from SAS 9.3 online doc: If you place an equal sign between the ampersand and the macro variable name of a direct macro variable reference, the macro variable’s name displays in the log along with the macro variable’s value.

Hello Groovy in SAS 9.3

Sun, 28 Oct 2012 23:07:12 +0000

see, it’s hip to be square ‘cuz SAS has a new PROC that’s GROOVY -Chris Hemedinger, Poetry on our own terms These days I played Proc Groovy (new in SAS 9.3) for a while because Groovy natively supports JSON (JavaScript Object Notation) data format. I downloaded much JSON data in the past few weeks(Github archive for example). ——–I’m a line separator ———

Tony Barr and the Early History of SAS

Tue, 23 Oct 2012 22:32:17 +0000

SAS historians (not all SAS users!) know Anthony James Barr (aka Tony Barr or Jim Barr) was the designer and implementor of SAS language (1966-1979) and the first Chairman and President of SAS Institute (1976-1979). In this picture was Tony Barr in the first SUGI (January 26-28, 1976, Kissimmee, FL), taken by the first SUGI chair Julian Horwich. It’s said in 1970s, everyone had long hair! I was particularly interested in the early history of SAS when traced back the SAS file system to FFS (Formatted File System) in Tony Barr’s statement, SAS Related History.

DoW-Loop: A Quick Note from SESUG 2012

Sat, 20 Oct 2012 23:26:57 +0000

Once upon a midnight dreary, While I pondered, weak and weary, Over many a quaint and curious Volume of SAS-L galore, While I coded, nearly snapping, Suddenly there came a tapping As of someone gently wrapping Code around my mental core. “’Tis’ Do-Whitlock Loop”, I muttered, “Wrapped around my brain core” – This it was, and so much more!

Be Your Own SAS Admin: Create Customized SAS Sessions

Thu, 18 Oct 2012 19:51:36 +0000

You may find such SAS desktop shortcuts useful: click icon A then you get a SAS session titled “Project A” (you may notice your SAS session with title “SAS” by default). In this session, the current working folder is just where you put Project A (the default may be like C:usersyourID). Also, some autoexec codes run including libname, fileref, formats search, macro loading and other customized options/settings. Keep session A alive, then click icon B: you get your SAS session specified for Project B.

Incorporate SAS/IML to Base SAS?

Tue, 16 Oct 2012 22:28:41 +0000

Since SAS 9.3, ODS Graphics was moved into Base SAS and other statistical procedures, which means the SAS/Graph license is not needed anymore to access ODS Graphics facilities. It’s definitely nice, but from customers’ point of view, it is not critical necessary: since the “minimum set of SAS system” in most SAS sessions includes the Base SAS, SAS/Stat and SAS/Graph, there is almost no impact for SAS users to switch the license of ODS Graphics.

SAS Perl Regular Expression (PRX) Talk in the forthcoming SESUG 2012

Thu, 11 Oct 2012 21:20:39 +0000

Research Triangle will host SouthEast SAS Users Group (SESUG) 2012 conference (October 14-16, Sheraton Imperial Hotel, Durham, NC; back to the home of SAS!). It will be my first regional SAS user conference and I will have a talk on SAS Perl Regular Expression(PRX), in the morning of next Monday, Oct 15. SAS PRX is an old school topic somehow. After dumping the paper, I think I should restructure my talk in a more accessible way.

Powershell: Up to v3.0

Mon, 08 Oct 2012 22:22:23 +0000

Just update Windows Powershell to version 3.0 and I do like this new GUI (Integrated Scripting Environment, ISE) with a console, a script window and a command list (Powershell is a true Windows shell; you may try to type 1+2 in the old CMD console!): I also used a free Powershell GUI product, PowerGUI with similar interface like ISE. The latest PowerGUI (3.2.0.2237) supports Powershell 2.0 by default (while you can force it to play with PS 3.

Current Working Folder in SAS

Mon, 08 Oct 2012 22:19:47 +0000

The concept of current working directory in SAS is not as important as it sounds. Most SAS users point to file reference (fileref) and library reference (libname, a “namespace”-like mechanism to access files ), which are much convenient in large project with multiple folders. Sometimes I push files to the current folder in relatively small task because I don’t want to jump to C:\Users\jhu\appdata\Local\Temp\SAS\Temporary Files_TD2820_like folder to get the temporary datasets (in WORK library) hard code a folder to receive ODS outputs.

Two SAS AF Utilities: LogFilter and Format Viewer

Tue, 02 Oct 2012 14:40:05 +0000

Nice to have (and easy to follow, both tested in Windows 7 with SAS 9.3M1 while they were both written in lower version of SAS): 1. LogFilter: a SAS log checker by rtsl http://www.ratcliffe.co.uk/rest_logfilt.htm 2. A SAS format viewer by Frank Poppe http://frank-poppe.xs4all.nl/SAS/edit_format.html

Set Up R (for SAS Programmers)

Sun, 30 Sep 2012 21:19:31 +0000

Yep, I am a SAS programmer: during the day time, I use SAS for my work; at the evening, I use R for entertainment.— Charlie Huang Getting start to play with R (it is really a wonderful learning tool): 12. SAS IML Studio and R Integration (not tested) 11. SAS IML and R Integration (not tested) 10. SAS Base and R Integration: PROC R also a blog post.

Statistical Notes (5): Confidence Intervals for Difference Between Independent Binomial Proportions Using SAS

Sun, 23 Sep 2012 21:23:59 +0000

A guy notices a bunch of targets scattered over a barn wall, and in the center of each, in the “bulls-eye,” is a bullet hole. “Wow,” he says to the farmer, “that’s pretty good shooting. How’d you do it?” “Oh,” says the farmer, “it was easy. I painted the targets after I shot the holes.” – An Old Joke Note on how to get the difference between independent binomial proportions using SAS PROC FREQ (tested in SAS 9.

Where is SAS Output Anyway?

Sat, 22 Sep 2012 22:10:12 +0000

It is said in An Introduction to R, one of R official documents (Current Version: 2.15.1): There is an important difference in philosophy between S (and hence R) and the other main statistical systems. In S a statistical analysis is normally done as a series of steps, with intermediate results being stored in objects. Thus whereas SAS and SPSS will give copious output from a regression or discriminant analysis, R will give minimal output and store the results in a fit object for subsequent interrogation by further R functions.

New Book on Confidence Intervals for Proportions by Prof. Newcombe

Fri, 21 Sep 2012 21:18:45 +0000

Check it out: a new book on confidence intervals for proportions by Prof. Robert G. Newcombe just recently available from CRC Press: Confidence Intervals for Proportions and Related Measures of Effect Size This book also supplies a set of Excel spreadsheets to calculating CIs, see http://www.crcpress.com/product/isbn/9781439812785 Other users may be interested on a SAS or R implementation.

Frequentist or Baysian in A Binomial Test

Fri, 21 Sep 2012 01:20:10 +0000

A latest updated post on Freakonomics, Beware the Weasel Word “Statistical” in Statistical Significance!, seemed to attempt to challenge frequentist statistics by Bayesian. I have no research on Bayesian and won’t jump to the debates. I’d rather to use this case to apply the Dragon’s Teeth and Fleas logic of hypothesis testing (at least I think frequentist is pretty intuitive). A story goes with this post, that Bob won 20 times out of total 30 rounds with Alice in a children card game, “War”.

Statistical Notes (4): Dragon’s Teeth and Fleas: Hypothesis Testing in Plain English

Sun, 16 Sep 2012 14:12:07 +0000

Statisticians aren’t the problem for data science. The real problem is too many posers — Cathy O’Neil you actually do need to understand how to invert a matrix at some point in your life if you want to be a data scientist. — Cathy O’Neil I was asked in several different occasions to explain hypothesis testing to non-technical people in plain English. Now I think I got a pretty neat one while honors belong to three great Germans, Friedrich Engels, Karl Marx, and Heinrich Heine.

Statistical Notes (3): Confidence Intervals for Binomial Proportion Using SAS

Sat, 15 Sep 2012 00:13:03 +0000

A guy notices a bunch of targets scattered over a barn wall, and in the center of each, in the “bulls-eye,” is a bullet hole. “Wow,” he says to the farmer, “that’s pretty good shooting. How’d you do it?” “Oh,” says the farmer, “it was easy. I painted the targets after I shot the holes.” – An Old Joke Last year I dumped piece of SAS codes to compute confidence intervals for single proportion using 11 methods including 7 presented by one of Prof.

Statistical Notes (2): Equivalence Testing and TOST (Two One-Sided Test)

Wed, 12 Sep 2012 22:39:23 +0000

_Programmers Need to Learn Statistics Or I will Kill Them All –Zed A. Shaw_ In an equivalence testing example against lognormal data, a TOST (Two One-Sided Test) option used in SAS TTEST procedure: proc ttest data=auc dist=lognormal tost(0.8, 1.25); paired TestAUC*RefAUC; run; And the output: Since the 90% (who not 95%? see below) limit of the geometric mean ratio (0.9412) , [0.8634,1.0260], lies in the FDA endorsed equivalence bounds [0.

Statistical Notes (1): Geometric Mean and Geometric Mean Ratio

Wed, 12 Sep 2012 20:15:15 +0000

_Programmers Need to Learn Statistics Or I will Kill Them All –Zed A. Shaw_ Just read since SAS 9.2, the TTEST procedure also natively supports Equivalence Test by simply adding a TOST option (Two one-sided tests). In a example, TTEST procedure reports a geometric mean as 0.9412, which is the geometric mean of a ratio, TestAUC/RefAUC. It can also be calculated manually by proc sql; select exp(mean(log(TestAUC/RefAUC))) “geometric mean of TestAUC/RefAUC” from auc; quit;

Is There Any Better Way? Publishing Process For CDISC Standards Documentation

Thu, 19 Jul 2012 01:54:10 +0000

1. The Pain I read from Lex Jansen (@LexJansen) that CDISC SDTM v1.3 and SDTMIG v3.1.3 were newly released. It’s pretty nice since CDISC SDTM was supposed to be released semiannually in the new publishing cycle. We can see the team put great efforts on this new version, but frankly speaking, this delivery (the way to display, not the content itself) is far away elegant. The new SDTM Implementation Guide (IG) v1.

Tab is a Tab is a Tab is a Tab

Sat, 14 Jul 2012 23:10:40 +0000

Well I’m a big fan of “tabs” and feel much more comfortable when launching everything into tabs: web pages, text files, PDF files, Windows Office files (Word, Excel, PPT), and even folders. Followings are my best of choices to share: 1. File Explorer: Clover http://ejie.me/ There are lots of fancy/complicated file/folder management system beyond Windows Explore (Total Commander might be the most famous one) while Clover, as its name indicates, is just as simple as I need:

Sublime Text 2 for SAS Programmers: A Quick Note

Fri, 13 Jul 2012 00:27:57 +0000

I’m playing with a new text editor, Sublime Text 2, and it has much potentials to replace my current handy Notepad++ and VIM. A quick note for further exploration(will keep update). /**update: 2012-10-16, Roy Pardee maintains a nice bundle (here), so the following bundle will be not updated any more.*****/ Also created a Github repository to hold all my SAS configuration files for Sublime Text 2. Fork me at

Blogging is Awesome: CDISC Bloggers

Fri, 04 May 2012 22:42:32 +0000

I remember when blogging was cool. Before the specializing and monetizing and Twitter-izing. —Peter Dewolf Well I think blogging is still cool (and awesome_ and _awesome …). The most appealing personal reason is, blog posts are Google searchable and suitable for archive while Tweets NOT. Admittedly I hold some sort of Existentialism 2.0: if it is not Google searched, it doesn’t exit! Last month I placed a post on how to keep pace with CDISC from its official channels and I feel cool to add an appendix of source from the awesome blogosphere.

Digital Life and Personal Data Analysis

Wed, 04 Apr 2012 20:32:06 +0000

This picture (by newsobserver.com) took from Wake County Library Book Fair in North Carolina Fairground where I also went to pick up some books. I was and still is a big book (almost paper books) fan. But when I was standing among the 450,000 used books in the book fair, I felt depressed and reluctantly had an impression: the paper books will and should go away in the future.

OpenCDISC Validator V1.3: An Unboxing Review (1): counting issue

Sat, 31 Mar 2012 16:58:00 +0000

The lasted OpenCDISC Validator version 1.3 was released at 29 March, 2012 (btw, there is a typo in the Line 1 of CHANGELOG.txt within the package: “2012” not “2011”). As usual, you can submit the following SAS scripts to get some basic information(remember to customize your directory): filename CDISC url "https://raw.github.com/Jiangtang/Programming-SAS/master/Rules_Count_OpenCDISC_XML.sas"; %include CDISC; %Rules_Count_OpenCDISC_XML(dir=C:OpenCDISC1.3compareopencdisc-validator_1.3config) and you get a summary of validation rules of OpenCDISC Validator V1.

Fetch CDISC Control Terminology Files in NCI Vocabulary Repository: All in One Click

Fri, 30 Mar 2012 00:23:17 +0000

CDISC Control Terminology is the most frequent updated model among CDISC standards. Take SDTM as example, the latest SDTM terminologies released at 23 March 2012; and from 2009 to 2011, there were 15 different SDTM terminology versions! If you just rely on your own local repository, you might miss the pace somehow. Here is a simple approach. Just submit the following one line of codes in a shell, wget http://evs.

Quick Notes on RTP CDISC User’s Group Q1 Meeting

Wed, 28 Mar 2012 00:43:03 +0000

It’s my first time to attend a local event, RTP (Research Triangle Park) CDISC User’s Group meeting, Q1 and here are some quick notes. 1. people Almost fresh faces for me. It’s nice to meet Jack Shostak of Duke Clinical Research Institute again. I visited him in Duke last year after SAS Global Forum in Las Vegas. Jack has a forthcoming book on SAS and CDISC, Implementing CDISC Using SAS: An End-to-End Guide.

US Post Beats Menu Cost

Sat, 24 Mar 2012 01:29:47 +0000

One of the interesting observations in my first few months in US: there is no price printed in the new mail stamps! It is interesting because as a former student of economics, I think US Post system did a nice attempt to beat the so called “menu cost” which should honor to Harvard economist Mankiw. Literally the menu cost is the cost of a restaurant to print new menus due to price adjustment.

Not Documented, Not Exist?!

Wed, 29 Feb 2012 19:48:44 +0000

Weeks before I first got the column edit mode in SAS 9.3 Enhanced Editor. Since had no previous SAS versions in machine, I just checked the online documentation of SAS 9.2 in which it’s not mentioned (while documented in SAS 9.3) then I concluded the column mode is an exciting new feature in SAS 9.3: if it is not documented, it doesn’t exist! Then it turns that I was wrong. My friend Venkat Veerisetty just told me that this feature was available since SAS v8.

Happy Leap Year From Logical (Point of View) Operators

Wed, 29 Feb 2012 00:14:07 +0000

Today is Feb 29, –another leap year! I just found it is a good chance to raise the topic of logic operators again. To determine if 2012 is a leap year, use the straightforward logic expression (demo is SAS, follow the pseudo code in the wiki page of leap year): if mod(year,4) = 0 then do; if mod(year,100) = 0 then do; if mod(year,400) = 0 then is_leap = 1; else is_leap = 0; end; else is_leap = 1; end; else leap=0;

GitHub and Weekend Programming

Sun, 19 Feb 2012 16:17:32 +0000

Yihui of Iowa State just texted me that GitHub is programmers’ Facebook. Inspired by him(great thanks!), I also begin to play with GitHub now: https://github.com/Jiangtang Currently I only created one repo as personal SAS code repository. To kill weekend time, I uploaded piece of codes to count the OpenCDISC validation rules by models. To use it: filename CDISC url “https://raw.github.com/Jiangtang/Programming-SAS/master/Rules_Count_OpenCDISC_XML.sas”; %include CDISC; %Rules_Count_OpenCDISC_XML(dir=C:tempOpenCDISCsoftwareopencdisc-validatorconfig) while get:

Column Mode in SAS 9.3 Enhanced Editor

Sun, 12 Feb 2012 11:05:28 +0000

A Sunday finding, column mode new in SAS 9.3 Enhanced Editor (in Windows OS): Currently this column mode only supports selecting (highlighting) but not editing. Hoo, it’s better than nothing. Update: the column selection mode was available since SAS v8, see http://www.jiangtanghu.com/blog/2012/02/29/not-documented-not-exist/

Face Off: Review OpenCDISC XML files

Sat, 11 Feb 2012 16:35:35 +0000

OpenCDISC, the first open source CDISC validator, is already in the toolbox of FDA reviewers (CDER/CBER, see CDISC Standards in the Regulatory Submission Process, 26 January 2012, P.33). The key features in OpenCDISC is a dichotomy of validation rules (XML based) and application logic. Currently OpenCDISC Validator (Version 1.2.1) officially supports the four following CDISC modules: SDTM 3.1.2 SDTM 3.1.1 Define.xml 1.0 ADaM 1.0 You can get the corresponding configuration files (validation rules) online or in the software folder (in .

Happy New Year (Yes Again)

Tue, 24 Jan 2012 12:06:11 +0000

Then I feel great to reset my year of 2012 as brand new one. Today, Jan 23, is the first day of Chinese New Year, and it is Monday, the first day of work week–it is always joyful to have such coincidenceJ. HAPPY NEW YEAR! I had a big move this year. Actually I passed through multiply new years of 2012: I took my flight from Beijing in Jan 1 (happy new year!

Vim as A SAS IDE

Sun, 13 Nov 2011 11:11:14 +0000

Few configurations (just copy this sas.vim file to C:\Program Files\vim\vim73\syntax if you also use gVIM 7.3 at Windows) to make Vim as a simple SAS IDE where F3: run SAS codes (in batch mode) F4: close other two windows (the current active window is Log window after F3 running; F4 jump to SAS file with full window) F5: jump to SAS file F6: jump to Log file F7: jump to lst file (list output)

My Collection of SAS Macro Repositories

Tue, 08 Nov 2011 21:41:54 +0000

Then I just find that the most effective and safest way to synchronize bookmarks across machines is making them Google searchable, i.e, putting them online. Followings are my personal collections of SAS macro Repositories (I will keep it update according to new sites reached and your inputs). Most of them are rich, well documented and friendly for navigation and review: /*\*General**/ 1. SAS Macros by Richard DeVenezia**** http://www.devenezia.com/downloads/sas/macros/index.php Richard is a very active contributor in SAS-L.

Get Start with WPS, and Call for an Elegant SAS IDE!

Sat, 05 Nov 2011 21:50:42 +0000

I got a trial version of WPS (the latest version 2.5.2.0 at Windows), which engine can interpret “some of the language of SAS”. I took piece of codes for testing and some passed while some popped up with errors (so currently it is only a limited version of SAS). I don’t drive into the deep part yet of what WPS can do and can’t do, but I do love the WPS way to organize projects, folders and files (including souse files):

Hello Python

Mon, 31 Oct 2011 20:29:43 +0000

Inspired by Jian’s polyglot programming practice, I also begin to brush up Python and C++ which I learned during graduate school. Following is a Python response to one of Jian Dai’s former programming challenges for lines count of source codes: [cce lang=”python”] import os #count number of lines of #single file def lineCount(fileName): countSingle=0 for line in open(fileName): countSingle += 1 return countSingle #count number of lines of #directory and subdirectories

An Online Latin to English Translator via SAS

Sat, 08 Oct 2011 21:47:59 +0000

Last month I submitted piece of SAS codes for a monthly programming challenge hosted by Jian Dai to translate the Latin motto of Hogwarts School in Harry Potter into English__: draco dormiens nunquam titillandus You can get the meaning using Google search of course—but not in Google Translator (Google Translator can’t recognize all of such Latin words!). Jian posted a concise Perl way to parse webs which contain this Latin phrase and key words “mean”, “means” and such and you can always find page like

Map and Reduce in MapReduce: a SAS Illustration

Tue, 04 Oct 2011 21:31:18 +0000

In last post, I mentioned Hadoop, the open source implementation of Google’s MapReduce for parallelized processing of big data. In this long National Holiday, I read the original Google paper, _MapReduce: Simplified Data Processing on Large Clusters_ by Jeffrey Dean and Sanjay Ghemawat and got that the terminologies of “map” and “reduce” were basically borrowed from Lisp, an old functional language that I even didn’t play “hello world” with. For Python users, the idea of Map and Reduce is also very straightforward because the workhorse data structure in Python is just the list, a sequence of values that you can just imagine that they are the nodes(clusters, chunk servers, …) in a distributed system.

An Analytical Valley: Big Data and Data Scientists (and SAS Programmers)

Wed, 14 Sep 2011 22:15:17 +0000

Tom Davenport reported an observation that Silicon Valley is becoming more analytical since companies in the Valley such as Google, Facebook, eBay, LinkedLn all have strong presences in analytics. Besides such predominant companies, I’d also like to add Yahoo to the list although Yahoo is no longer in its peak. Yahoo is the largest sponsor and contributor of Hadoop, an open source framework for distributed processing of so called “big data”.

Fours Errors in SAS 9.2 Fisher’s Iris Data in SASHELP Library

Sat, 03 Sep 2011 21:13:42 +0000

In the previous post, I just mentioned that Fisher’s Iris Data is embedded officially in SASHELP library in SAS 9.2. Note that even in SAS 9.1.3, you can also find this data with several instances from some demos in user guide (just search “Iris” in “SAS Help and Documentation” accompany with you SAS product), for example, in SAS 9.1.3 IML.

Iris dataset is so important and popular that researchers round the world use it as benchmark to test and compare their algorithms and also as pedagogical purpose. It is also the overwhelming No. 1 dataset considering popularity in UCI Machine Learning Repository. Here 4 errors in SASHELP.iris listed for your consideration if interested and if you find some slightly differences in outputs following some demos out of SAS using this data:

Error 1: Line 35, the PetalWidth of Setosa should be 2 mm, not 1 mm;

Error 2: Line 38, the SepalWidth of Setosa should be 36 mm, not 31 mm;

Error 3: Line 38, the PetalLength of Setosa should be 14 mm, not 15 mm;

Error 4: Line 119, the PetalLength of Virginica should be 69 mm, not 70 mm.

For errors 1-3, there is also an interesting story in statistical literature. In 1936, Fisher the Great published his famous paper, _The use of multiple measurements in taxonomic problems_ and the Iris data also attached (called Fisher Version in this post). In the following years (until today), people cited this paper and the Iris data Fisher Version is also replicated and distributed worldwide and then a version with above errors 1-3 might gain a very dominant popularity (I don’t know the source of there errors). In UCI Machine Learning Repository, the dataset iris.data is the one with such 3 errors (called UCI Version as well).

We could see that the duplicated UCI Version is even more popular in some extension than its original Fisher Version (SASHELP.iris also seems to be copied from UCI Version). Story goes on. In 1998, James Bezdek and other scholars just found the three discrepancies between Iris Fisher Version and UCI Version (and in some published papers using the same version of data). You can read it in _Will the Real Iris Data Please Stand Up?_

Bezdek then proposed to use the original Fisher Version of Iris, and UCI Machine Learning Repository also documented these three errors and added new dataset called bezdekIris.data (Bezdek Version) which is exactly Fisher Version (iris.data kept and I think it is because now the so called error version is also valuable).

Return to error 4 and I can’t figure out why and I might as well call it Iris SAS Version. Note that the unit in SAS Version is millimeter (mm), while others version all use centimeter (cm).

The interesting part is that I also check the Iris data in SAS 9.1.3 IML mentioned before and not surprising, it is exactly the Fisher Version (you can also find a right one in a demo from SAS 9.2 IML Studio 3.2).

The following codes generate several Iris versions:

iris_uci: UCI Version with both CM and MM as unit

bezdekiris_uci: Bezdek Version or Fisher Version with both CM and MM as unit

iris_mm: UCI Version with MM as unit and attributes alike SASHELP.iris, SAS Version

bezdekiris_mm: Bezdek Version or Fisher Version with MM as unit and attributes alike SASHELP.iris, SAS Version

Who is Alfred?

Wed, 31 Aug 2011 22:18:09 +0000

Tell me something about Alfred, male or female? age? height and weight? Oracle database (version 9 and below) had a well known default demo account SCOTT with a password, TIGER (and TIGER was the name of the real person Bruce Scott ’s cat, see) and in this account, there are some tables named DEPT, EMP, BONUS and SALGRADE (you can read their meaning). Almost every Oracle DBA learn SQL using these database and an joke just says that in DBA’s meetings, people just warm up saying “how about Smith?

I am a 20% SAS Nerd!

Tue, 23 Aug 2011 21:57:14 +0000

Kirk Paul Lafler drafted a checking list for identifying a SAS nerd (or geek, in its positive ways) in one of his intriguing papers: _You Could be a SAS® Nerd If …_ Here I’m glad to find that I am roughly a 20% SAS nerd (12 matched in all 57 lists): You blog SAS-related comments and technical solutions frequently. You have more than five SAS blogs in your RSS feed.

SAS Bloggers in Action (2): Jian Dai and his SAS Academy

Mon, 15 Aug 2011 20:59:29 +0000

My first post on SAS bloggers begins with Rick Wicklin and I plan a series of posts. It would be nice for a statement of rational before following pages. As a learner, I benefit frequently from lots of high quality blogs. But I’m also slightly lazy as a reader: I read almost all blogs in Google Reader rather than returning to the original web pages and adding some comments. Here I’d like to post a series of SAS bloggers and their blogs (that I subscribed and read frequently) to extent my thanks, share with friends and sometimes amuse the bloggers themselves:)

Retrieve blogs using SAS

Wed, 20 Jul 2011 21:47:46 +0000

Recently I posted a frequency analysis on Rick Wicklin’s popular SAS/IML blog. Sanjay Matange also produced a nice heatmap on Rick’s blogging history using the summary data I published. Here just release the ideas and SAS codes to get data from Rick’s blog dynamically. You may modify the codes slightly to obtain data from all other SAS in-house blogs (http://blogs.sas.com/index.php) since they share the same template. For other blogs, you should research the web pages accordingly to get the best suitable methods and this post can also serve as an example.

First step: define the scope

For my purpose, I only need the titles and publish dates of Rick’s posts. It is so called the “metadata” of the blog. I do not need all the post contents. By the way, if all information needed, you can use a blog backup tool, or write codes to retrieve all the pages of http://blogs.sas.com/iml at the maximum depth, or simply, you can write to Rick and say: hey Rick, could you please send me all the contents of your blog? And Rick may go to the management console of his own blog, export all the contents to an XML file and get back to you.

Second step: analyze the web pages

Browse to the right panel of Rick’s blog, in the ARCHIVES frame, click “Older”:

And you get

This page just gives a big picture of Rick’s blog (ARCHIVE section is always a good place to get such metadata, for example, archives for my blog). But we need more. Click “view topics” for example of September, 2010:

This page is exactly what we want with titles and dates. Open an editor to write codes immediately to read all the information in this page?—wait. Currently this blog has posts across 11 months and you can expect the increase. You should design a dynamic method to read all the topics pages: Sep 2010, Oct 2010, … and, today().

Return to the archives page. RCM (right click your mouse) and select “View page source” if you use Google Chrome web browser (“View Source” in IE; “View Page Source” in Firefox) and you get all the HTML scripts (Note: you DO not need any knowledge of HTML to understand this post). Copy and paste them into a text editor supporting HTML syntax highlighting (such as Notepad++). Search all instances of “view topics” we mentioned before:

We are lucky. They are 11 instances of “view topics” accompanying with 11 hyperlinks for the currently 11 months’ archives of Rick’s blog. We can read such 11 hyperlinks to a macro array for dynamic retrieval.

Then we return to the single topics page, for example of September, 2010. Review the HTML source file. Search for “posted_by_date” and we get 14 instances which is same as the number of posts in September 2010:

We should also need to locate all the instances of titles. Search “/iml/index.php?/archives/” and we get 17 responses:

We see 3 instances at end of the finding results don’t contain any titles. You can check other pages to confirm such pattern. Yes we can use regular expressions to parse the HTML pages to locate more exactly for the titles. But for a quick job and due to the relative simple HTML pages, some basic SAS character functions are enough for our purpose. In the following codes, limited regular expressions are used only to remove HTML tags such as “

After such explorative search of HTML scripts, we can get the basic idea where can we find the interested information. Then we begin to coding work.

Third step: Coding at last!

For our purpose, we should first read the archive page to get all the topics links to a macro array, then read the all the topics pages dynamically. Finally, we should also add the all the calendar dates with holidays. Some friends may find that they met piece of the following codes before. Yes, such codes just assembled some skills what I learned from Art Carpenter, Richard DeVenezia, Jian Dai and lots of programmers before!

3-1: read archive page

%let URL=http://blogs.sas.com/iml/index.php?/archive;

filename archive URL “&URL”;

data archive;
length text $1024;
infile archive lrecl=1024;
input text $;
text= infile;
if index(text, “>view topics<“) then output;
run;

data archive1;
set archive;
summary=scan(text,4,’”‘);
run;

3-2: read all topics pages

data null;
set archive1 end=eof;
I+1;
II=left(put(I,2.));
call symputx(‘summary’||II,summary);
if eof then call symputx(‘total’,II);
run;

%macro readit;
%do i=1 %to &total;
filename f&i URL “&&summary&i”;

        data f&i;
length text $1024;
infile f&i lrecl=1024;
input text $;
text= infile;
if index(text, “/iml/index.php?/archives/”) or index(text, “posted_by_date”) then output;
run;

    /remove HTML tags;/
data ff&i;
set f&i;
prx=prxparse(“s/<.*?>//”);
call prxchange(prx,99,text);
drop prx;

        flag=ifn(mod(n,2),1,2);
grpn=&i;
if index(text,”201″) and length(text)<10 then delete;/be carefull! hard coding;/
        seq=n;
run;

     /transpose data;/
data fff&i;
set ff&i;
by grpn seq flag;

        retain title;
if first.flag then title=lag(text);
if flag=1 then delete;
run;

    %end;

%mend readit;
%readit

%macro getall;
data Rick;
set %do i=1 %to &total; fff&i %end; ;
seq=seq/2;
drop flag;
run;
%mend getall;
%getall

data rick2;
set rick;

    datetime=scan(text,2,”,”);
year=scan(text,2,”.”);
month=scan(scan(text,2,”,”),1);
day=scan(scan(text,2,”,”),2);
week=scan(text,6);
dt=compress(catx(“”,day,substr(month,1,3),year));
worddat=input(dt,date9.);
format worddat ddmmyy10.;

    m=scan(put(worddat,ddmmyy10.),2);
my=compress(catx(“”,year,m));
run;

proc sort ;
by worddat descending seq ;
run;

SAS Bloggers In Action(1): Rick Wicklin, SAS/IML and “Color Revolution”

Sun, 17 Jul 2011 16:16:49 +0000

It is well known that the French writer, author of The Three Musketeer, Alexandre Dumas, wrote his master piece of work in different colored papers according to literary genre: non-fiction on rose, fiction on blue, poetry on yellow The SAS blog writer, author of Statistical Programming with SAS/IML Software, Rick Wicklin of SAS Institute, also leads a strong “color revolution” in SAS blog community: In an interesting personal statement, Blogging, Programming, and Johari Windows, Rick summarizes his rich and colorful blogging rhythms according to the above Johari window:

Tango Haiku

Fri, 15 Jul 2011 22:50:18 +0000

Happy weekend and Haiku again! My finger hurts, I can not dance Tango in Bastille. So… any ideas? … … … … … … OK. I am a terrible Haiku writer. The sentences are not self explained without /*commentaries*/ . So the story behind the scene… One day my remote workstation was very very slow due to some network issues. But I still needed to write and run SAS codes in the server which is located in France.

A SAS Implementation of Confidence Intervals for Single Proportion: Eleven Methods

Thu, 14 Jul 2011 22:05:44 +0000

The following two papers by Professor Robert Newcombe, in my limit observation, are the most frequently cited papers in the industry for CI calculation: Two-sided confidence intervals for the single proportion: comparison of seven methods. Newcombe RG, Stat Med , Volume 17 , 8 (April 1998) pp.857-872 Interval estimation for the difference between independent proportions: comparison of eleven methods. Newcombe RG, Stat Med , Volume 17 , 8 (April 1998) pp.

Dive into CDISC Express (5): Generate and Validate SDTM domains and define.xml

Thu, 07 Jul 2011 22:32:03 +0000

Dive into CDISC Express (1): Introductory

Dive into CDISC Express (2): Create a New Study

Dive into CDISC Express (3): Navigate mapping file

_Dive into CDISC Express (4): Data manipulation techniques_

A more friendly PDF version of these all CDISC Express series is also available in

http://jiangtanghu.com/docs/en/CDISCExpress.pdf

The following tasks, such as generating SDTM domains and define.xml, need just some clicking button work in CDISC Express using a well designed mapping file. Few words needed due to the software.

Dive into CDISC Express (4): Data manipulation techniques

Mon, 04 Jul 2011 20:52:42 +0000

Dive into CDISC Express (1): Introductory

_Dive into CDISC Express (2): Create a New Study_

_Dive into CDISC Express (3): Navigate mapping file_

4.3 Data manipulation techniques in CDISC Express

CDISC Express supplies relative rich sets of data manipulation techniques assembling with SAS languages used for data mapping. Following is a not limited listing and I will keep it updated.

4.3.1 Reference one dataset

A raw dataset name appear in “Dataset” column indicate a “set” operation in SAS.

All dataset options can be used when referencing a dataset, such as

siteinv(drop=invcode)

siteinv(rename=(invcode=inv))

siteinv(where=(invcode ne “”))

You can also reference an external dataset. You should incorporate the external file in spreadsheet with name beginning with an underscore, “_”, and “_visits” in this case:

Then you can use it in any domains needed, e.g., TV domain:

There is a macro %cpd_importlist used to import the external dataset, “_visits”. Again, this macro roots in C:Program FilesCDISC Expressmacrosfunction_library.

Using a macro call to re-sharp or modify an input dataset offers great flexibility referencing data. We will also discuss the benefits later on.

4.3.2 Assignment

You can assign a number, string and a dataset variable with any valid SAS functions to a SDTM domain variable in “Expression” column.

Sometimes a temporary variable needed for later calculation. You can produce such temporary variable in “Dataset” column with an assignment in the “Expression” column just similar with any other domain variables. Two differences: first, such temporary variables named begin with an asterisk, “*”; second, all temporary variables will not be included in the final domain. Once created, such temporary variables can be used for any other expressions.

There are three special symbols used in “Dataset” column of CDISC Express. Asterisk, “*” indicates a temporary variable, while other two are

Tilde, “~” : indicate a variable used for supplemental domain (SUPPQUAL).

Number sign, “#”: indicate a variable used for comments domain (CO).

Another symbol, at sign, “@”, used in “Expression” column, indicated referencing a variables produced before:

In this case, “AGEU” uses “AGE” as input, while “AGE” is calculated before. “@AGE” just indicates the dependency. In concept, it looks like the “calculated” option in SAS PROC SQL:

proc sql ;

select (AvgHigh – 32) * 5/9 as HighC ,

(AvgLow – 32) * 5/9 as LowC ,

(calculated HighC – calculated LowC)

as Range

from temps;

quit;

4.3.3 Match-merging

We already got a math-merging example before. If “all” appears as a dataset in the “Dataset” column, all the previous datasets should be merged first for later processing by the common key specified in “Merge Key” column. If no key assigned, patient ID is used by the system.

CDISC Express also supports two types of join, inner join and outer join (left, right, full) using data steps. The implementation has slightly difference with standard SQL, but the ideas are same.

We add a new column, “Join”, usually beside the “Merge Key” column.

There are two values for “Join”, “O” or “I” while “O” stands for “outer join” and “I”, “inner join”. A join indicator “I” equals a dataset option “in=” in action while “O” means no. Use the above as illustration, the corresponding SAS codes behind look like

data temp;

merge demog(in=a) siteinv(in=b);

by sitecode;

if b;

run;

This is so called “right outer join”. The combination of “I” and “O” in these two datasets can perform all the four types of join, one inner join and three outer join:

As we could see, if no “Join” column specified, CDISC Express will perform inner join by default.

So far CDISC Express cannot support multiply merge keys. For example, the following file is illegal currently:

Dataset	Merge Key
arm	siteid, grpno
armdescri	siteid, grpno

The developer Romain indicated that such enhancements would be raised to the next round of product road map and he also proposed a work around. To use multiple keys for merging, we can create a temporary variable holding such multiple keys as a concatenation then this temporary variable can be used as a single merging key.

4.3.4 Concatenating

Above we discussed lots about “merge” operation in CDISC Express. This section dedicated for “set” operation. We already know how to “set” one dataset for referencing, but how to “set” multiple datasets, i.e, “Concatenating”?

Symmetrically, an “all” appears in “Dataset” column indicating merging operation, an “all (stack)” indicates concatenating operation:

The above file can be also translated to SAS codes for better understanding:

Dive into CDISC Express (3): Navigate mapping file

Sun, 03 Jul 2011 21:19:18 +0000

Dive into CDISC Express (1): Introductory

Dive into CDISC Express (2): Create a New Study

4. Step 2 of 6: Generate mapping file

Generating template (blank) mapping file only needs pieces of effort by submitting generate_mapping_template.sas. The toughest one is to fill it with mapping rules according to specified study.

4.1 Get the blank template mapping file (generate_mapping_template.sas)

To get the blank template mapping file, just fill the one line of macro call in generate_mapping_template.sas:

%createmapping(filespec=SDTM_Specs_3_1_1.xls, Dom=CM AE TV, req=YES, perm=YES, exp=YES);

Also, you can specify SDTM implementation version, 3.1.1 or 3.1.2. For domains (&Dom), DM, CO and SUPPQUAL will be created automatically; you should list others accordingly:

SDTM 3.1.1: SV CM EX AE DS MH DV EG IE LB PE QS SC VS TV TI XD SU XR XS XE TR (Total: 22)

SDTM 3.1.2: AE CE CM DA DS DV EG EX FA IE LB MB MH MS PC PE PP QS SC SE SU SV TA TE TI TS TV VS (Total: 28)

You should also choose the “CORE” variable (REQUIRED, PERMISSIBLE and EXPECTED) by triggering &req, &perm, and &exp to “YES” or “NO”. Note that

REQUIRED and EXPECTED variables must always be included (req=YES, exp=YES);

PERMISSIBLE variables included if needed (perm=YES or perm=NO)

Submit generate_mapping_template.sas and you can get a blank template mapping file tmpmapping.xls in C:Program FilesCDISC Expresstemp.

Copy it to C:Program FilesCDISC ExpressstudiesCLINCAPdocMapping file – working version for example used for study “CLINCAP” and then fill all the blank columns (it NEEDS efforts!).

If this mapping file passes the validation process, a final version named mapping.xls will be copied automatically to C:Program FilesCDISC ExpressstudiesCLINCAPdocMapping file – validated version for later processing.

Note that if you already have some validated mapping file for other studies, it would serve as a good start rather than using the blank template from the scratch.

4.2 Navigate mapping file

Let’s take a look at the “real” worked mapping file for a demo study first, in C:Program FilesCDISC Expressstudiesexample1docMapping file – working version.

The first sheet is a welcome dashboard:

Then StudyMetadata sheet, a XML metadata specification used to generate define.xml. you need only add some information in “Values” column:

The FORMAT sheet:

Such format structure is similar with the one we export the format from a format catalog using

proc format library=library cntlout=format_out;

run;

In most production environment, programmers get formats from clinical data management group. If the entire formats are assigned into proper libraries (work or library), you don’t need to export such formats into this spreadsheet. Of course in the format sheet, you can type some customized format.

A typical domain sheet (AT LAST!) that needs efforts and our understanding of the software, DM for example:

From the ‘Dataset’ column, three raw datasets from C:Program FilesCDISC Expressstudiesexample1source needed to map into DM domain, demog, siteinv and eligassess. Note that you can use any data step options such as drop=, rename=, where= for the input datasets.

At the last of ‘Dataset’ column, “all” indicates that all the previous datasets mentioned above should be merged together for final processing.

In the ‘Merge Key’ column, ‘sitecode’ is designed to datasets demog and siteinv which means demog and siteinv should be merged by the common key, ‘sitecode’.

As we mentioned, all the previous datasets should be merged at last. But there is no common key settled in the ‘Merge Key’ column. It is a common rule: if no key specified for merge, USUBJID is used by default.

The third column is ‘CDISC variable’, which list all the needed variables according to implementation version. An important note: you do not need to implement all the variables according to the order as they appear in the blank template mapping file. In the previous blank file, “AGE” in DM domain is ordered in Line 12, but in this working file, “AGE” is calculated in the second last order. The variable order of final DM domain will be as same as the blank one.

It makes sense in practice. For example, the sequential variable, e.g. AESEQ is ordered after USUBJID, but you can only get the sequential number when all other variables well done. So SEQ variables are always computed in the final stage in a working mapping file.

“Expression” column specify the mapping rule from raw datasets to SDTM domains. Assignments, expressions and macro calls (rooted in C:Program FilesCDISC Expressmacrosfunction_library) are allowed in this column and most of them are straightforward. We will discuss more in the following section.

Sum up, we can “translate” this mapping sheet to SAS codes for better understanding of CDISC Express architecture:

Dive into CDISC Express (2): Create a New Study

Sat, 02 Jul 2011 21:21:22 +0000

Dive into CDISC Express (1): Introductory 3. Step 1 of 6: Create a new study (create_new_study.sas) Open create_new_study.sas in C:Program FilesCDISC Expressprograms, you can see only one line of a macro call: %addnewstudy(studyname=my new study); Just assign a study name to the macro variable, &studyname, e.g, “CLINCAP”: %addnewstudy(studyname= CLINCAP); Submit the codes, you can find a folder named “CLINCAP” with the same structure as the two demo studies imbedded in this application(example1 and example2) in C:Program FilesCDISC Expressstudies, see(the left and right panels are folders and files before and after the execution of create_new_study.

ADJECTIVE Encounters

Sat, 02 Jul 2011 10:51:57 +0000

Really she is the strangest creature in the world, far from heroic, variable as a weathercock, “bashful, insolent; chaste, lustful; prating, silent; laborious, delicate; ingenious, heavy; melancholic, pleasant; lying, true; knowing, ignorant; liberal, covetous, and prodigal”— in short, so complex, so indefinite, … –Virginia Woolf, The Common Reader, First Series (1925) I don’t know if Ernest Hemingway is still one of the recognized dominant writers in colleges English education.

Dive into CDISC Express (1): Introductory

Tue, 28 Jun 2011 21:24:33 +0000

Recently I did for my personal project some research on Clinovo’s open source application, CDISC Express, a SAS application based on Excel framework designed to map clinical data to CDISC SDTM domains automatically. Not perfect yet, but it is easily understandable and practically usable after few hours’ of exploration of user guide. And most important, it is on the right way: an automatic CDISC converter is the magic weapon in almost every clinical programmer’s dream.

SAS Global Forum 2011: Personal Notes

Sat, 04 Jun 2011 16:05:46 +0000

It’s wonderful. It’s awful. It’s wonderfully awful and awfully wonderful. –Frommer’s Las Vegas 2010 June at last! June, July and August (great thanks to French colleagues!) would be the best time to prepare a paper for next year’s SAS Global Forum (SGF, 2012 in Orlando, Florida). SGF2012 will hold at the “real” Walt Disney at Orlando against Las Vegas, the adults’ Walt Disney for SGF2011. Lots of children will be anticipated to register as guests for this meeting and I may contribute one:).

CDISC Express: A Glance

Mon, 16 May 2011 00:48:31 +0000

This weekend I tested an application that can automatically transform clinical data to CDISC SDTM compliant datasets(3.1.1 and 3.1.2), CDISC Express of Clinovo. According to its license statements, you can download it for free and for personal use only. The core of CDISC Express is an Excel configuration file called Mapping File which defines all the metadata and mapping rules required by CDISC SDTM standard. Then a set of SAS macros is used to

Sail for Las Vegas!

Sat, 19 Mar 2011 21:53:48 +0000

look to the master, follow the master, walk with the master, see through the master, become the master. The SAS Global Forum 2011 (Apr 4-7, Las Vegas) papers are now available on line: http://support.sas.com/resources/papers/proceedings11/TOC.html There are lots of Web 2.0 elements in this year’s SGF, including a Twitter hashtag #SASGF11, a Facebook account, a blog, a LinkedIn group, and of course, sasCommunity.org. I will give a talk at this conference (Tuesday, April 5, 05:30- 05:50 PM, Promenade Level, Neopolitan IV) to discuss how to get informed with little efforts from such multiple channels for SAS programmers using Gmail and Good Reader to create a personal information portal.

Too Big to Be Accurate(1): Which is the Most Powerful Calculator in the World?

Sat, 22 Jan 2011 17:05:17 +0000

Calculate the factorial of 171 (171!)? Just TRY! It is equal to 171*170*169*….2*1.

1. Google calculator

As Google fanatics, I first try to search the answer via Google:

Whoops, nothing interested returned! Type “170!” and get the output:

Why kinda things happened in this calculator? 171! is just equal to 171*170!.

2. Excel

Switch to Excel spreadsheet. Function fact(*) used:

Oo, interesting. The same.

3. SAS

Google and Excel may be the niche players in calculators’ family. Why not try to use some programming languages?

Feature Selection: Collections for Self Study

Sat, 15 Jan 2011 10:58:44 +0000

Recently I start to learn the algorithms and applications of feature selection. The term “Feature”, wildly used in machine learning and data mining literatures, simply means “Variable”. In some practices, for example, a neural network model uses a decision tree as input; the tree performs the function of variables selection. The Arizona State University is maintaining a repository of feature selection, including original documentations, Matlab packages and user guide for the following popular algorithms so far:

Decision Trees in SAS Enterprise Miner and SPSS Clementine

Tue, 04 Jan 2011 20:44:50 +0000

Decision trees are included in SAS Enterprise Miner(EM). The counterpart is SPSS Clementine, which should be called IBM SPSS Modeler for precision after IBM’s acquisition of SPSS. Recently I read a paper on the comparisons of SAS EM, SPSS Clementine and IBM Intelligent Miner on their decision tree and cluster technology: _Decision Tree Induction & Clustering Techniques in SAS Enterprise Miner, SPSS Clementine, and IBM Intelligent Miner – A Comparative Analysis_ by Abdullah M.

SAS Data Step’s Built-in Loop: An illustrated Example

Mon, 03 Jan 2011 17:29:40 +0000

Some newbie SAS programmers take SAS as their first programming language even learned. Sometimes they are confused by the concept of “data step’s built-in loop” even after reading the well-written The Little SAS Book: A Primer: DATA steps also have an underlying structure, an implicit, built-in loop. You don’t tell SAS to execute this loop: SAS does it automatically. Memorize this: DATA steps execute line by line and observation by observation.

SGF: Caesars Palace in Las Vegas again

Thu, 30 Dec 2010 18:13:08 +0000

The mechanic, who wishes to do his work well, must first sharpen his tools. --Confucian Analects. BOOK XV.WEI LING KUNG.CHAP.IX. My paper _Work Smarter than Harder-tools for growing up a SAS programmer_ was accepted by SAS Global Forum 2011. It would be my first time to attend SAS user group conference worldwide. The draft version is available at http://jiangtanghu.com/docs/en/SGF2011_JiangtangHU(draft).pdf.pdf”) Welcome for any comments. The SGF2011 will be held at Caesars Palace in Las Vegas, Nevada.

A SAS Programmer’s End Year Haiku

Thu, 30 Dec 2010 16:18:20 +0000

End year Haiku again! Yesterday I finished my project, wrote a Haiku to colleagues worldwide to say happy new year, then closed my desktop, said goodbye to colleagues still in office: In December my baby born my project deliveries followed on Now my computers shutdown do until(6 Jan 2011) I will be OOO Please take blessings from John Happy New Year!

Blogging SAS

Tue, 30 Nov 2010 21:16:52 +0000

Almost at the same time, there are two SAS blogs aggregators popping up to the web for SAS programmers worldwide, one in Chinese, the other, English: http://saslist.com/ in Chinese, maintained by sxlion, also the owner of a SAS information site, http://saslist.net/ http://sas-x.com/ in English, maintained by Tal Galili, also the owner of R blogs aggregator, http://r-bloggers.com/ I try to express in a very symmetrical way. What’s more (and interesting), these two aggregators share a same WordPress template.

Recursion: Biblical Evidences

Sat, 06 Nov 2010 09:58:56 +0000

To Iterate is Human, to Recurse, Divine. –by L. Peter Deutsch, creator of Ghostscript. (I also love his saying “_The best thing about a Boolean is even if you are wrong, you are only off by a bit_”) I learned this quote from Tony Barr, designer and developer of SAS (the Statistical Analysis System). He also used the Recursive Descent method to develop SAS complier. Now Tony is working for AMOR (A Model of Reality).

Power of Logic Operators: a trick

Thu, 04 Nov 2010 07:09:12 +0000

Suppose you should group people based on their ages as follows: ID Age agegrp 001 1 1 002 4 2 003 5 2 004 5 2 005 2 1 006 4 2 007 5 2 008 2 1 009 9 3 010 8 3 and the rules:

Recursive Referencing and Binomial Proportion Interval

Wed, 03 Nov 2010 20:47:44 +0000

To understand recursion, one of the most important concepts in programming languages, you could watch the movie, Chris Nolan’s Inception: it is about a dream within a dream within a dream, …and, read two statistical papers by Professor Robert Newcombe, one of the most prolific statisticians: Two-sided confidence intervals for the single proportion: comparison of seven methods. Newcombe RG, Stat Med , Volume 17 , 8 (April 1998) pp.

Play Matrix within SAS(1): basic files processing

Fri, 29 Oct 2010 21:02:31 +0000

Recently I read Rick Wicklin’s IML blog with great interests(and anticipation for his fore-coming IML book, Statistical Programming with SAS/IML Software). SAS programmers have the following programming tools to facilitate their daily work: SAS data step: the basic SAS; a generation IV programming language, similar with other procedural languages such as C. SAS Proc SQL: SAS’s implementation of standard SQL(SQL-92). SAS IML(Interactive Matrix Language): SAS’s matrix manipulation language(like R and Matlab).

SAS Algorithmically(1): Newton-Raphson method

Thu, 21 Oct 2010 21:39:42 +0000

A good reference for the basic algorithms of Newton-Raphson method to calculate the square root of a number, see http://mathforum.org/library/drmath/view/52644.html And the SAS codes(self-explanatory): data root; /question: find the square root of 4/ x=4; /first choose a rough approximation of sqrt(4); actually, you can start with any numbers/ y0=1; count=0;/init count number/ do until (waccumulate count number/ y=(y0+x/y0)/2; /Newton’s formula/ w=abs(y-y0); /if close, exit;/ y0=y; /* otherwise, keep the new one*/ end;

Happy SAS Graphing!

Sun, 17 Oct 2010 20:02:33 +0000

I’m not a R user. Instead, I’m an observer. For example, I love the R Graph Gallery: http://addictedtor.free.fr/graphiques/thumbs.php As a SAS programmer, I also love Robert Allison‘s SAS/Graph Examples and Michael Friendly‘s SAS Graphic Programs and Macros (the only two SAS graph galleries available on website maintained by users): http://robslink.com/SAS/Home.htm http://www.datavis.ca/sasmac/ I used to be a casual user of SAS Graph. I’d like to add Graph into my toolbox in the following month: half of it due to the encouragement of the two SAS graph gurus, the other half SAS 9.

Logics in mathematics and in daily life: a statistical programming example

Fri, 08 Oct 2010 22:47:00 +0000

Refresh some basic logical propositions (or statements): implication: if P then Q (P—>Q) inverse: if not P then not Q (-P—>-Q) converse: if Q then P (Q—>P) contrapositive: if not Q then not P (-Q—>-P) contradition: if P then not Q (P—>-Q) Mathematically or logically speaking, if the implication statement holds, then the contrapositive holds, but the inverse does not hold, i.e., if P then Q, then we can get if not Q then not P, but we can not get if not P then not Q.

XML and SAS

Thu, 07 Oct 2010 11:44:38 +0000

Last month, I gave a talk, XML: the SAS Approach, in CDISC Interchange China 2010(at the Medical School of Fudan University, Shanghai, 2010-09-15). FDA favors CDISC and HL7, the two XML based standards, and SAS programmers in biopharmaceutical industry need incorporate the XML technology into their toolboxes. Fortunately, you don’t need to be an XML expert then to play XML in your daily work, and, SAS system DOES offer multiple tools and applications to handle XML files, i.

On three statistical realms

Wed, 06 Oct 2010 11:37:51 +0000

Peter Petocz and Anna Reid(2010) grouped three levels of students’ conceptions of statistics: Level I: focus on techniques Level II: focus on using data Level III: focus on meaning I found the three conceptions could be easily interpreted as the three kinds of state of learning and using statistics based on my personal experience: State I: focus on techniques—As a student of Economics and (then) Software Engineering, I needed some statistics techniques to support my study on data mining and machine learning.

From A Logical Point of View

Mon, 21 Dec 2009 23:53:47 +0000

This blog, “From A Logical Point of View”, is not supposed to be owned by a logician. Actually, the book, From A Logical Point of View, is a collection of logical and philosophical essays by W.V.Quine(1908-2000), an American philosopher and mathematician. A story about this book. From A Logical Point of View, was a calypso song by Robert Mitchum, an US actor, composer and singer. Quine enjoyed this music and used it as his new book, which is Quine’s best seller.

Work With Oracle: A Quick Sheet for SAS Programmers

Fri, 04 Dec 2009 09:33:00 +0000

(Note: All the followings are tested on Windows XP environment.) 0. Install Oracle Database 10g Express Edition Fast (and free) to download, easy to deploy and simple to admin–for learning and testing purpose, Oracle Database 10g Express Edition (Oracle Database XE, a mini version of Oracle Database 10g Release 2) are strongly recommended: 0.1 Download it at its homepage(206MB); 0.2 Install it following default settings; 0.3 Unlock accounts for HR and SCOTT.

Run data mining codes following William Potts

Fri, 20 Mar 2009 11:41:00 +0000

FYI: SAS Enterprise Miner and SAS Text Miner Procedures: Reference for SAS 9.1.3, see: http://support.sas.com/documentation/onlinedoc/miner/emtmsas913/listing.html This entry DOES exist in the SAS Support website, but it can’t be found by any search engine or documentation tree view. You’re recommended to download these files immediately due to SAS’s easy-dead hyperlinks.^-^ ps.SAS Institute provides no support for the use of Enterprise Miner and Text Miner Procedures when they are invoked directly, outside of the Enterprise Miner graphical user interface.

Free Machine Learning Courses (Stanford) in YouTube

Mon, 02 Mar 2009 10:28:00 +0000

FYI: http://www.youtube.com/view_play_list?p=A89DCFA6ADACE599

SAS User Books and Data Mining Software Comparision: Quick Links

Tue, 17 Feb 2009 14:55:00 +0000

SAS Books Catalog(Jan, 2009) Data Mining Software 2009: Succesul Analyses at Affordable Prices(Nov. 2008, by mayato)

Basel II on Incremental Risk Charge(IRC): Quick Links

Thu, 12 Feb 2009 09:37:00 +0000

Basel Committee on Banking Supervision(Jan, 2009). Guidelines for computing capital for incremental default risk in the trading book. Consultative document. January. IRC Comments by RiskMetrics

FYI: Dashboard using SAS/Graph

Tue, 13 Jan 2009 15:01:00 +0000

see, http://support.sas.com/kb/26/134.html Slider chart indicator: SAS program version stored process version portlet version Slider chart dashboard: SAS program version stored process version portlet version Bullet graph indicator: SAS program version stored process version portlet version Bullet graph dashboard: SAS program version stored process version portlet version Dial meter indicator: SAS program version stored process version portlet version Dial meter dashboard: SAS program version stored process version portlet version Bar chart indicator: SAS program version stored process version portlet version Bar chart dashboard: SAS program version stored process version portlet version Telesales dashboard: SAS program version stored process version portlet version Web marketing analysis dashboard: SAS program version stored process version portlet version You would also like, Santa’s Dashboard:

R or SAS: Quick Links to the Recent Debates

Tue, 13 Jan 2009 10:37:00 +0000

Original post, 7 Jan, 2009 New York Times, Data Analysts Captivated by R’s Power byAshlee Vance. Key Point The popularity of R at universities could threaten SAS Institute. A Controversial Review by Anee Milley from SAS We have customers who build engines for aircraft.

Options Pricing Using SAS

Tue, 06 Jan 2009 16:20:00 +0000

There are some new financial functions in SAS9.2 Base, including 8 options pricing functions(formerly in SAS Risk Dimension). These functions can compute the price of both call and put options on different underlying assets (stock, futures, currency, and exchange asset), using the following models respectively: Black-Scholes model，the traditional stock options pricing model, see Fischer Black and Myron Scholes (1973) ; Black model，extension of Black-Scholes model on futures option (also called Black-76 model)，see Fischer Black(1976);

Happy New Year

Thu, 01 Jan 2009 23:45:00 +0000

新年快乐 ( Xin Nian Kuai Le), Guten Rutsch ins neue Jahr, Bonne Année, Nav varsh ki shubh kamnayey, Felice Anno Nuovo, Feliz Año Nuevo, Feliz Ano Novo, 明けましておめでとうございます (Akemashite Omedetô), Gelukkig Nieuwjaar, Szczęśliwego Nowego Roku, Καλή Χρονιά (Kalí Chroniá), Seh Heh Bok Mani Bat Uh Seyo, חג חנוכה שמח (Hag Hanukkah Sameah), Cчастливого Нового Года, <p> Happy New Year, </p>

Links: Risk Intelligence Vendors Review: 2008

Thu, 25 Dec 2008 16:26:00 +0000

You can get the big picture viewing different sources(REMEMBER: A vendor’s research methodology is as important as its rating): <div> </div> <div> <span style="font-family: Arial;">Chartis RiskTech 100 (October 2008)</span> </div> <div> <span style="font-family: Arial;"><a href="http://www.chartisresearch.com/showreport.asp?id=46">http://www.chartisresearch.com/showreport.asp?id=46</a></p> <p> <a href="http://2.bp.blogspot.com/_qlX7zyUQhog/SVNELKlQf2I/AAAAAAAABBk/O9jIagazNHU/s1600-h/RiskTech100-2008.PNG" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img id="BLOGGER_PHOTO_ID_5283641746445598562" style="cursor: pointer; width: 320px; height: 290px;" src="http://2.bp.blogspot.com/_qlX7zyUQhog/SVNELKlQf2I/AAAAAAAABBk/O9jIagazNHU/s320/RiskTech100-2008.PNG" border="0" alt="" /></a> </p> <p> </span></div> <div> <span style="font-family: Arial; font-size: 85%;"> </span> </div> <div> <img src="cid:01d201c9666a$7afda6e0$22121bac@apac.sas.com" border="0" alt="" hspace="0" align="baseline" /> </div> <div> <span style="font-family: Arial;"> </span> </div> <div> <span style="font-family: Arial;"><br /> FinTech100(2008)</span> </div> <div> <span style="font-family: Arial;"><a href="http://www.

Links–BI Industry 2008: Review and Prospect

Wed, 24 Dec 2008 17:42:00 +0000

/*Thanks the hints supplied by: A look back at 2008 and some crystal ball predictions…, byTammi Kay Geroge, from SASBlog*/ Major Data Warehousing Events of 2008 (and Predictions for 2009), by Michael Schiff, from TDWI, Major Data Warehousing Events of 2008: Everyone had an appliance story Industry consolidations continued The recessionary environment encourage further BI developments Open source grew Predictions for 2009: Further industry consolidation(Informatica by HP, SPSS by SAP) Cloud computing will come down to earth Open source growth will accelerate The IT world will become greener Major emphasis on solutions rather than tools and technology BusinessIntelligence Tools: Year in Review，by Cindi Howson, fromBeyeNetwork Top Virtualization Trends for 2009, by John Suit, from ZDNet Surround the Warehouse: Prediction for 2009 , by Neil Raden, from IntelligentEnterprise

Industry Review: SAS and Teradata Partnership

Fri, 19 Dec 2008 10:44:00 +0000

SAS and Teradata Partnership: Press Leading Companies See Value in SAS and Teradata Partnership SAS and Teradata Unveil Advantage Program to Bring Powerful In-Database Solutions and Services to Customers SAS and Teradata Enter into Strategic Partnership In BI industry, the pure players such as SAS, Teradata and Microstrategy, need to demonstrate their indispensable values against the megavendors, IBM (acquired Cognos), SAP (acquired Business Object), Oracle (acquired Hyperion) and Microsoft.

Santa and SAS Again: Santa’s Dashboard

Thu, 18 Dec 2008 09:34:00 +0000

(santaDashBoard.png, With permission by Mr. Robert Allison) Merry Christmas again. SAS marketing staff started up an interesting Christmas campaign on how Santa operates his workshop. Here is another wonderful work about Santa’s Dashboard, created by SAS senior R&D staff, Robert Allison. Robert is a master of graphics and visualization. You can view his SAS/Graph examples in the following link: http://robslink.com/SAS/Home.htm

The Making of an Analyst: A Supplement to What Makes a Good Business Analyst

Wed, 17 Dec 2008 09:40:00 +0000

I once commented the entry, What Makes a Good Business Analyst by Rajan Chandras, with an easy tone, If You Can Make it Here, You Can Make it Anywhere. The standards of a good analyst conclude by Rajan, in my opinion, are somewhat of very high bars. <div> <span style="font-family: Arial;"> </span> </div> <div> <span style="font-family: Arial;">In the recent SASCOM Magazine, Ted Cuzzillo published a relatively moderate enty, say, <em><a href="http://www.

Learn Time Series Analysis: Free Materials for SAS Users

Tue, 16 Dec 2008 16:07:00 +0000

0. A gentle Introduction to Time Series Analysis, may serve as fast learning materials: http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc4.htm <div> <span style="font-family: Arial; font-size: 100%;">1. An open source book(with data and code), <a href="http://statistik.mathematik.uni-wuerzburg.de/timeseries/index.php?id=preamble"><em>A First Course on Time Series Analysis: examples with SAS</em>, by Prof. Michael Falk</a>, is available in:</span><span style="font-size: 100%;"><br /> </span> </div> <div> <span style="font-family: Arial; font-size: 100%;"><a href="http://statistik.mathematik.uni-wuerzburg.de/timeseries/"></p> <blockquote> <p> http://statistik.

Links of 2008-12-16: Financial Engineering, Ponzi Scheme, SAS PC Game

Tue, 16 Dec 2008 09:34:00 +0000

The State of Financial Engineering Ponzi Scheme Returns: SEC Charges Bernard L. Madoff for Multi-Billion Dollar Ponzi Scheme WolfenSAS: A PC Game written by SAS Code

Delivers the Right Toys and Goodies to the Right Boys and Girls: Story of Santa and SAS

Sun, 14 Dec 2008 19:57:00 +0000

How to know the boys and girls’ real demands around the world? and how to predict their demands in next Christmas? How to purchase toys and goodies with a balance of costs and profits? and how to deliver them more efficiently? There are lots of questions in the list of Santa, CEO of Santa’s Workshop. SAS’s marketing staff held a very creative champion for the coming Christmas. You can watch the interview with Santa in youtube.

If You Can Make it Here, You Can Make it Anywhere: On What Makes a Good Business Analyst by Rajan Chandras

Sun, 14 Dec 2008 00:24:00 +0000

In the latest post, What Makes a Good Business Analyst?, Rajan Chandras cites some soft items from Forrester’s Business Analyst Assessment Workbook: Ability to think abstractly, identify patterns, and generate ideas and solutions Understanding of when and how to escalate issues or needs Understanding of and ability to delivery the appropriate level of detail needed for each task Interest in exploring and understanding new concepts and topic areas Emotionally invested in the work Ability to learn by shadowing stakeholders Ability to clearly articulate technology in terms stakeholders can understand Understanding of the organizational culture and its impact on processes and projects (this one seems obvious, but the latter phrase is more profound than might seem at first glance) Ability to drive a decision analysis and selection process Ability to recognize patterns in requirements and categorize them appropriately What’s more, there are some suggestions by Rajan Chandras himself:

Data Mining in Stock Market

Sat, 13 Dec 2008 23:55:00 +0000

Data Mining in Stock Market? Is it crazy? or is it just a hopeless try? Every mentor in mathematics and finance educates us that the stock market is too chaotic and sentimental to use mathematical models. Most of all gift rock scientists are concentrated in the study of interest of rates and fixed income securities. It sounds profitable to use mathematical and statistical models to predict the price of stock, but there are little successfull stories.

Haiku from SAS R&D staff

Wed, 10 Dec 2008 11:37:00 +0000

First prompts are silent. Subsequent prompts loud and clear. Now all prompts are heard. Poem from R&D staff? Yes. Rhyming sonnets were shakespeare-like complex; they wrote Japanese haiku, showed as above. The SAS R&D staff should complete some paper work in defects system before changing a code. They use informal descriptive language(HAIKUUU!) in the early stage. Chris Hemedinger, a senior software engineer at SAS, collected some haikus in his blog to show the humor side of SAS R&D staff.

Happy grow up

Wed, 10 Dec 2008 10:21:00 +0000

Happy hearts and happy faces, Happy play in grassy places– –Good and Bad Children by Robert Louis Stevenson I read this verse in W. Bennentt’s popular book, The Book of Virtues, during the bus-to-company time this morning. It’s interesting to read Stevenson’s Treasure Island, of course in Chinese edition when I was young. <div> <span style="font-family: Arial; font-size: 85%;"> </span> </div> <div> <span style="font-family: Arial;">Yes, it sounds “uncool”, –I went to work, with technical documents in my bag, and read a for-children book.