OWASP O2 Platform Blog

Fortify FVDL files – Creating and consuming the schema and CSharp file

An o2 user send me a nice 400Mb Fortify file to see if O2 could do something with it (since Fortify’s GUI was not able to load it).

The next couple posts will document how to go from an XML file to creating an API to consume and to visualize it.

For this demo I will use the Fortify FVDL files that where published by SATE 2008 results (part of NIST SAMATE project) which Fortify participated. You can download the entire SATE 2008 data from their website (which includes the findings files from all participats and SATE’s result), or you can download just the Fortify FVDL (and xsd) from O2’s S3 repository: Fortify-Sate-2008.zip (the scripts below will use this files, but if you have access to *.fvdl files, you can use them)

To start, open an ‘O2 Quick Development Environment GUI’

Then download the demos files and unzip them to a local temp folder:

var demoFileUrl = "http://s3.amazonaws.com/Demo_Files/Fortify-Sate-2008.zip";
var localFile = demoFileUrl.uri().download();

Next step is to unzip the file:

var localFile = "".tempDir().pathCombine("Fortify-Sate-2008.zip");
if (localFile.fileExists().isFalse())
{
    var demoFileUrl = "http://s3.amazonaws.com/Demo_Files/Fortify-Sate-2008.zip";
    demoFileUrl.uri().download();
}   
return localFile.unzip_File();

Then add a check to only unzip if the target folder doesn’t exist and package it in a Lambda method which can be easily consumed by the main script:

Func<string> getFolderWithFvdlDemoFiles =
    ()=>{
            var localFile = "".tempDir().pathCombine("Fortify-Sate-2008.zip");
            if (localFile.fileExists().isFalse())
            {
                var demoFileUrl = "http://s3.amazonaws.com/Demo_Files/Fortify-Sate-2008.zip";
                demoFileUrl.uri().download();
            }    
            var targetFolder = @"..\_Fortify-Sate-2008".tempDir(false).fullPath(); // by default this willl resolve to C:\O2\_tempDir\_Fortify-Sate-2008
            if (targetFolder.dirExists().isFalse())
                localFile.unzip_File(targetFolder);            
            return targetFolder;
        };

var folderWithFvdlFiles = getFolderWithFvdlDemoFiles();       
return folderWithFvdlFiles.files(true,"*.fvdl");

There a number of *.fvdl files available, and since the first thing we need is to create an *.xsd for them, let’s pick the smaller one (in this case naim.fvdl)

var folderWithFvdlFiles = getFolderWithFvdlDemoFiles();        
var naimFvdl = folderWithFvdlFiles.files(true,"naim.fvdl").first();
return naimFvdl;

Usually at this stage we can use the O2 Fluent XML Apis to do this. Basically, in most cases this works:

var folderWithFvdlFiles = getFolderWithFvdlDemoFiles();        
var naimFvdl = folderWithFvdlFiles.files(true,"naim.fvdl").first();


return naimFvdl.xmlCreateXSD(); // creates XSD from Xml file

Unfortunately, for this XML file, we get the error “The table (Node) cannot be the child table to itself in nested relations.” (which is quite a common error when creating XSDs from XML files)

The 2nd usual attempt is to try to use the XSD.exe tool that comes with Visual Studio, But that just returns the same error:

The 3rd attempt is to use VisualStudio 2010 for the conversion. Before we open the file in visual studio, we will need to create a new copy (or rename it) with an .xml extension (ie C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.xml)

 Then from the XML menu, select the ‘Create Schema’ menu Item

Which this time around should work 🙂 :

Save this file locally

Now go back to the O2 development Gui (or open up a new one), and lets create the CSharp file. Start by making sure you have a correct reference to the file:

var xsdFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.xsd";
return xsdFile.fileExists();

call the xsdCreateCSharpFile method to create a CSharp file:

var xsdFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.xsd";
return xsdFile.xsdCreateCSharpFile(); 

This will create a file called ‘C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.cs’, which you can take look by opening it in a source code editor control

var topPanel = panel.clear().add_Panel();
var xsdFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.xsd";
var csharpFile = xsdFile.xsdCreateCSharpFile(); 
var codeEditor = topPanel.add_SourceCodeEditor(); // add source code editor to topPanel
codeEditor.invoke("setMaxLoadSize","1000");       // invoke the private method that sets the max size of the file to load using the SharpDevelop control (if bigger the file is opended in a listview)
codeEditor.open(csharpFile);                        // open file

if try to compile this csharp file, you will get a number of “The type of namespace ‘xyz’ could not be found”  errors

This is caused because the CSharp file was created using the Link2Xml Apis which are included in the O2_Misc_Microsoft_MPL_Libs.dll.

The solution is to add this as a reference at the top of the file, which will make the compilation work

Note that this extra reference could had also be introduced programatically like this:

var xsdFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.xsd";
var csharpFile = xsdFile.xsdCreateCSharpFile(); 

csharpFile.fileContents()
      .insertBefore("//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll".line())
      .saveAs(csharpFile);

The final step, now that we have the CSharp file (‘C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.cs’), is to use it as a reference and load up the original Xml file using it.

To make it clear, lets do this on a clean O2 Development GUI Environment

If you look at the CSHarp file you should notice that there is namespace

namespace xmlns.www.fortifysoftware.com.schema.fvdl {

…which contains a FVDL class

public partial class FVDL : XTypedElement, IXMetaData {

…which contains a Load static method

        public static FVDL Load(string xmlFile) {
            return XTypedServices.Load<FVDL>(xmlFile);
        }

This is the method that we will need to call , so that we can get a strongly typed version of our XML file

var xmlFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl";
var fvdl = FVDL.Load(xmlFile);
return fvdl;

//using xmlns.www.fortifysoftware.com.schema.fvdl
//O2File:C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.cs
//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll

 … which finally gives us the object that we want:

…and provides strongly-typed access to the fvdl data, including code-complete support

  ->

As a first exampe,  here is the list of vulnerability objects in the loaded xml file:

Note: one way to see in more details the type of objects that are availble is to call the {object}.details() method:

var xmlFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl";
var fvdl = FVDL.Load(xmlFile);
var vulnerabilities = fvdl.Vulnerabilities.Vulnerability; 
vulnerabilities.details();

..which opens a popup window that can be navigated by properties or fields (note that the data is only loaded on selection)

Going back into Vulnerability filtering …

… where it gets really powerful is when we use Linq queries to filter the data:

var xmlFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl";
var fvdl = FVDL.Load(xmlFile);
var vulnerabilities = fvdl.Vulnerabilities.Vulnerability; 

return (from vulnerability in vulnerabilities
        select vulnerability.ClassInfo.Kingdom).Distinct(); 

In this case, here is a distinct list of the vunerability’s Kingdoms

…  or a quick consolidated view of  the vulnerability data:

var topPanel = panel.clear().add_Panel();
var xmlFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl";
var fvdl = FVDL.Load(xmlFile);
var vulnerabilities = fvdl.Vulnerabilities.Vulnerability; </pre>
&nbsp;

var results =  (from vulnerability in vulnerabilities
                  select new  {
                                kingdom = vulnerability.ClassInfo.Kingdom,
                                analyzer = vulnerability.ClassInfo.AnalyzerName,
                                classId = vulnerability.ClassInfo.ClassID,
                                defaultSeverity = vulnerability.ClassInfo.DefaultSeverity,
                                instanceId = vulnerability.InstanceInfo.InstanceID,
                                instanceSeverity = vulnerability.InstanceInfo.InstanceSeverity,
                                confidence = vulnerability.InstanceInfo.Confidence,
                                function = vulnerability.AnalysisInfo.Unified.Context.Function.name,
                                file = vulnerability.AnalysisInfo.Unified.Context.FunctionDeclarationSourceLocation.path,
                                line = vulnerability.AnalysisInfo.Unified.Context.FunctionDeclarationSourceLocation.line,
                                col = vulnerability.AnalysisInfo.Unified.Context.FunctionDeclarationSourceLocation.colStart
                               
                            });
return results;

 

This  ‘Linq result’ object, is a local C# anonymous class, and is better seen and analyed on a TableList:

var topPanel = panel.clear().add_Panel();
var xmlFile = @"C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl";
var fvdl = FVDL.Load(xmlFile);
var vulnerabilities = fvdl.Vulnerabilities.Vulnerability; </pre>
&nbsp;

var results =  (from vulnerability in vulnerabilities
                  select new  {
                                kingdom = vulnerability.ClassInfo.Kingdom,
                                analyzer = vulnerability.ClassInfo.AnalyzerName,
                                classId = vulnerability.ClassInfo.ClassID,
                                defaultSeverity = vulnerability.ClassInfo.DefaultSeverity,
                                instanceId = vulnerability.InstanceInfo.InstanceID,
                                instanceSeverity = vulnerability.InstanceInfo.InstanceSeverity,
                                confidence = vulnerability.InstanceInfo.Confidence,
                                function = vulnerability.AnalysisInfo.Unified.Context.Function.name,
                                file = vulnerability.AnalysisInfo.Unified.Context.FunctionDeclarationSourceLocation.path,
                                line = vulnerability.AnalysisInfo.Unified.Context.FunctionDeclarationSourceLocation.line,
                                col = vulnerability.AnalysisInfo.Unified.Context.FunctionDeclarationSourceLocation.colStart
                               
                            }).toList();

topPanel.add_TableList("Showing {0} Vulnerabilties".format(results.size()))
        .show(results);                           
return "done";

//using xmlns.www.fortifysoftware.com.schema.fvdl
//O2File:C:\O2\_tempDir\_Fortify-Sate-2008\Fortify-Sate-2008\sate2008-Fvdl\naim.fvdl.cs
//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll

That wraps up this example, the next related entries will continue from here 🙂

July 16, 2011 Posted by | Fortify, Interoperability | 3 Comments

Using ‘Amazon S3 Browser’ script to upload files to Amazon S3

If you have an Amazon S3 account (http://aws.amazon.com/s3/) then you might find useful to use the ‘Amazon S3 Browser.h2’ script to upload files to the S3 servers (you can also create folders and delete files)

To upload files, select the target foldder (top left treeview) and  just drop the desired file on the the bottom left treeview (the red color indicates that the file is being uploaded)

When the upload has completed, the bottom treeview will refresh and show the uploaded file there:

To get the link to the file, click on it and copy it from the top right textbox

For small files (like images or html files) you can get a quick preview by selecting the ‘Auto Preview Selected File’ checkbox

Note that when you upload files using this GUI/Script the file permission will be set to public (the default is to make it private).

The ‘Amazon S3 Browser.h2‘ script (shown below) consumes the API_AmazonS3.cs file and is mainly focused on building the GUI:

//O2Ref:WatiN.Core.1x.dll
var topPanel = O2Gui.open<Panel>("Amazon S3 Browser",700,400);
var autoPreviewSelectedFile = false;
var amazonS3 = new API_AmazonS3();  </pre>
&nbsp;

topPanel.add_Label("Logging in to S3", 150,300);
amazonS3.login();
topPanel.clear();
var statusLabel = topPanel.parentForm().add_StatusStrip(); 
statusLabel.set_Text("Welcome to O2's Amazon S3 Browser (click on folder to see files)");
var foldersTreeView = topPanel.add_TreeView()
                           .showSelection();
                          
var previewArea = foldersTreeView.insert_Right<Panel>(topPanel.width() /2 ).add_1x1("Selected File Url", "Selected File Contents",false, 65) ;
var previewPanel = previewArea[1];
var selectFile = previewArea[0].add_TextBox().fill();
previewArea[0].add_CheckBox("Auto Preview selected File (to see single files just press enter above)",40,4, (value) => autoPreviewSelectedFile = value).autoSize(); 

var filesTreeView = foldersTreeView.insert_Below<TreeView>(topPanel.height()/2)
                                   .showSelection();  

MethodInvoker refreshFolders =
    ()=>{
            foldersTreeView.clear();
            foldersTreeView.add_Nodes(amazonS3.folders());
        };
       
MethodInvoker refreshFileView =
    ()=>{
            amazonS3.showFilesInTreeView(filesTreeView, foldersTreeView.selected().get_Text());           
        };       
                                  
foldersTreeView.insert_Below<Panel>(25)  
               .add_LabelAndTextAndButton("New folder name:", "", "create",
                           (newFolderName)=>
                               {
                                statusLabel.set_Text("Creating folder:{0}".format(newFolderName));
                                   if (amazonS3.create_Folder(newFolderName).valid())
                                   {
                                       foldersTreeView.clear();                                      
                                    foldersTreeView.add_Nodes(amazonS3.folders());   
                                   };
                               });
filesTreeView.onDrop(
    (fileOrFolder)=>{
                        if (fileOrFolder.isFile())
                        {
                            filesTreeView.backColor(Color.LightPink);
                            O2Thread.mtaThread(
                                ()=>{
                                        statusLabel.set_Text("adding file:{0}".format(fileOrFolder));
                                        var currentFolder = foldersTreeView.selected().get_Text();
                                        amazonS3.add_S3_Object(currentFolder, fileOrFolder);
                                        amazonS3.showFilesInTreeView(filesTreeView, currentFolder);                           
                                        statusLabel.set_Text("Done");
                                        filesTreeView.backColor(Color.White);
                                    });
                        }
                           
                    });
  
var foldersContextMenu = foldersTreeView.add_ContextMenu();
var filesContextMenu = filesTreeView.add_ContextMenu();

//menu actions 
foldersContextMenu.add_MenuItem("Delete Folder", 
                            ()=>{
                                    var folderToDelete = foldersTreeView.selected().get_Text();
                                    if ("Are you sure you want to Delete the folder: {0}".format(folderToDelete).askUserQuestion())
                                        if (amazonS3.delete_Folder(folderToDelete).valid())                                      
                                        {
                                            statusLabel.set_Text("Deleting folder:{0}".format(folderToDelete));
                                               refreshFolders();
                                           }
                                });
 
filesContextMenu.add_MenuItem("Delete File", 
                            ()=>{
                                    //filesTreeView.SelectedNode =
                                    var currentFolder = foldersTreeView.selected().get_Text();
                                    var fileToDelete = filesTreeView.selected().get_Text();
                                    if ("Are you sure you want to Delete the file: {0}".format(fileToDelete).askUserQuestion())                                   
                                    {
                                        statusLabel.set_Text("Deleting File:{0}".format(fileToDelete));
                                        if (amazonS3.delete_File(currentFolder, fileToDelete).valid())                                      
                                            amazonS3.showFilesInTreeView(filesTreeView, currentFolder);
                                    }
                                });

// events
foldersTreeView.afterSelect<string>(
    (folder) =>{
                    statusLabel.set_Text("Showing files in folder: {0}".format(folder));
                    refreshFileView();
                });

filesTreeView.afterSelect<Uri>(
    (uri)=>    {               
                var url = uri.str();
                "url: {0}".info(url);
                statusLabel.set_Text("Showing details for file: {0} ".format(url));
                selectFile.set_Text(url);               
                if (autoPreviewSelectedFile)
                    amazonS3.showFileInControl(previewPanel, uri);                                       
            });

selectFile.onEnter(
    (url)=> {               
                amazonS3.showFileInControl(previewPanel, url.uri());       
             });

// ContextMenu

filesContextMenu.add_MenuItem("Refresh", refreshFileView);

var folders = amazonS3.folders();
foldersTreeView.add_Nodes(folders);   
amazonS3.showFilesInTreeView(filesTreeView,"");

return "done";
//using O2.External.IE.ExtensionMethods

//O2File:API_AmazonS3.cs
//O2Ref:AWSSDK.dll
//O2Ref:O2_External_IE.dll
//O2Ref:System.Xml.dll
//O2Ref:System.Xml.Linq.dll

 

July 16, 2011 Posted by | EC2, Interoperability, WatiN | Leave a comment

Humm … .NET Random class is not Thread Safe?

I was doing some stress testing on a app (using O2), and suddenly my test data was all ‘aaaaaaa’ (vs random letters). After some debbuging it seems that that problem is that the .NET Random class is NOT thread safe. This as quite a lot of security implications and is defenitely not what I was expecting (see references at the bottom for links that confirm this behaviour).

Basically what happens is that if a random object is accessed from multiple threads, eventually it gets corrupted and starts returning 0 (zero).

For example, O2 has a couple helper classes to return random values:

//return 10.randomLetters();      // EXtjJkudEI
//return 20.randomLetters();      // rkjdQCFBzhCASriyhpzG
//return 10.randomNumbers();    // 1440637687
//return 20.randomNumbers();    // 70412467593222068252
//return 10.randomString();        // m8}XEYhy#G
//return 20.randomString();        // \('VVx%0f0,63>Ar35'!
//return 10.random();            // 4
//return 1000.random();            // 831
//return 100000.random();        // 72394
return 1000000.random();        // 261239

The problem happened that on my multi-thread tests sudently the 10.randomLetters() method returned ‘aaaaaaaaaa’ vs returning a random string with 10 letters.

If you look at the O2 API file with those Extension Methods (in Crypto_ExtensionMethods.cs) you see can how they work. The scenario where you will get all ‘a’ in this method will happen when  random.NextDouble() returns 0. As I was coding this API I have to say that I never expected this to happen (i.e. even assuming that there could be some colisions, I never expected the Random class to completely stop working!)

To find this issue, I first isolated the use of the randomLetters() method using this script:

Action startLoop =    
    ()=>{
        var loop  = 10;
        var count = 100000;
        for(var i = 0 ; i < loop; i++)
        {
            var startTime = DateTime.Now;
                       
            for(var j =0 ; j < count ; j++)
                10.randomLetters();
            var timeSpan = DateTime.Now - startTime;
            "[{5}/{6}] added {0} in {1}:{2} : {3} : {4}".info(count,timeSpan.Seconds, timeSpan.Milliseconds , 2.randomLetters(), 10.randomLetters(), i, loop);
           
            if (2.randomLetters() == "aa")
            {
                "PROB!!".error();
                return;               
            }
        }
    };
O2Thread.mtaThread(()=> startLoop());   
O2Thread.mtaThread(()=> startLoop());    
return 15.randomLetters();

This is the execution result:

And in order to find out the exact root cause (and to make sure it was not a problem with my code), I wrote the following PoC to replicate the problem using new instances of the the Random class :

var randomObject = new Random((int)DateTime.Now.Ticks);  
var sync = new ManualResetEvent(false);</pre>
&nbsp;

var threadCount = 5;
var innerLoopMax = 400000;
var totalCounter = 0;       

Action<int> startLoop =    
    (id)=> {
                sync.WaitOne();               
                "Running Loop for: {0}".info(id);
                       
                for(var j =0 ; j < innerLoopMax ; j++)           
                {                                       
                    if (randomObject.Next() == 0 )
                    {
                        "PROB!! : {0} : {1}".error(j, totalCounter);
                        return;               
                    }
                    totalCounter++;
                }
                "done: {0}".info(id);                       
            };
           

for(var i = 0 ; i < threadCount ; i++)            
    O2Thread.mtaThread(()=> startLoop(i));    

sync.Set();

//using System.Threading

run #1:


 
run #2

 

run #3

 

References:

It seems that others have noted this, and the answer eventually comes down to ‘the Random is not Thread Safe …. just live with it”:

This issue could introduce quite suble security vulnerabilites, specially in multithreaded environments like Asp.Net

July 16, 2011 Posted by | .NET | Leave a comment