OWASP O2 Platform Blog

FxCop Security rules : A nice to have feature on top of O2 platform

While we have seen the  effect of   static analysis at run time in the community, which  was first described with a PoC  using Microsoft’s static analysis tool CAT.NET   and OWASP O2 Platform on top o Visual Studio .Net, we strongly believe  on the value added of using O2 platform as a part of our development work.

In this same way,  a really nice to have feature would be an integration with FxCop . Namely,  FxCop analyzes managed code  and reports information about those assemblies. It analyzes several  areas including : COM, Design, Globalization,  Naming, Performance, Security and Usage. 

The latest version of FxCop was  included as a part of the Microsoft Windows SDK for Windows 7 and .NET Framework 4

Here you have some useful links with all the information  about this tool:

  1. For downloading it:
    http://blogs.msdn.com/b/codeanalysis/archive/2010/07/26/fxcop-10-0-is-available.aspx
  2. The  ISO files can also be  downloaded from the below link:
    http://www.microsoft.com/en-us/download/details.aspx?id=8442
  3. FxCop ASP.NET Security Rules : This is a really interesting project hosted at CodePlex and it offers  a set of rules for ASP.NET applications:
  4. http://fxcopaspnetsecurity.codeplex.com/

It would be great to have those security  rules available on O2 Platform and why not via real-time analysis :).

FxCop ASP.NET Security rules

FxCop ASP.NET Security rules

July 19, 2012 Posted by | .NET, Tools | , , | 1 Comment

Exporting Checkmarx SAST Database into XML files

Once I added VistaDB support to O2, I was able to export the CWE content from CheckMarx database into TeamMentor  (see videos at PoC of integrating TeamMentor with Checkmarx),

This post covers the part where we export the entire Checkmark database into flat XML files (for the script that consumed the CWE.xml see Case Study – Creating a CWE Library from CheckMarx data )

Viewing table data: Configurations

var vistaDB = new API_VistaDB();
vistaDB.ConnectionString = @"data source='C:\Program Files\Checkmarx\Checkmarx Application Server\CxDB.vdb3'";

var dataTable = vistaDB.table("Configurations").dataTable();
panel.clear().add_DataGridView().dataSource(dataTable);

//O2File:API_VistaDB.cs

Viewing table data: CWE

var vistaDB = new API_VistaDB();
var dataTable = vistaDB.table("CWE").dataTable();
panel.clear().add_DataGridView().dataSource(dataTable);

//O2File:API_VistaDB.cs

Get table as XML

var vistaDB = new API_VistaDB();
var xml = vistaDB.table("Configurations").xml();
return xml;

//O2File:API_VistaDB.cs
//O2Tag_DontAddExtraO2Files.

View table in DataGridView

var vistaDB = new API_VistaDB();

var dataTable = vistaDB.table("Configurations").dataTable();
panel.clear().add_DataGridView().dataSource(dataTable);

//O2File:API_VistaDB.cs
//O2Tag_DontAddExtraO2Files

Saving one table as XML file

var vistaDB = new API_VistaDB();
var checkMark_XmlDumps = "_CheckMark_XmlDumps".tempDir(false);
var tables = vistaDB.tables();

var table = tables[0];
var targetFile = checkMark_XmlDumps.pathCombine(table.Name + ".xml");

return table.xml().saveAs(targetFile);

//O2File:API_VistaDB.cs
//O2Tag_DontAddExtraO2Files

Saving all tables

var vistaDB = new API_VistaDB();
var checkMark_XmlDumps = "_CheckMark_XmlDumps".tempDir(false);
foreach(var table in vistaDB.tables())
{
 "saving table: {0}".info(table);
 var targetFile = checkMark_XmlDumps.pathCombine(table.Name + ".xml");
 table.xml().saveAs(targetFile);
}
return "done";

//O2File:API_VistaDB.cs
//O2Tag_DontAddExtraO2Files

Consuming  Saved CWE.xml

var checkMark_XmlDumps = "_CheckMark_XmlDumps".tempDir(false);
var cwe = checkMark_XmlDumps.pathCombine("CWE.xml").xRoot();

return cwe.elements().size();

Consuming saved CWE.Xml via cache: (faster)

var checkMark_XmlDumps = "_CheckMark_XmlDumps".tempDir(false);var cwe = "cweData".o2Cache<XElement>(()=> checkMark_XmlDumps.pathCombine("CWE.xml").xRoot());
return cwe.elements().size();
Tool to visualize the CWE data as HTML pages
var vistaDB = new API_VistaDB();
var checkMark_XmlDumps = "_CheckMark_XmlDumps".tempDir(false);
var xRoot = "cweData".o2Cache<XElement>(()=> checkMark_XmlDumps.pathCombine("CWE.xml").xRoot());

var topPanel = panel.clear().add_Panel();
var webBrowser = topPanel.add_WebBrowser_Control();
var treeView= webBrowser.insert_Left(200).add_TreeView();
var codeViewer = webBrowser.insert_Below().add_SourceCodeViewer();

treeView.afterSelect<string>(
 (text) =>
 {
 webBrowser.open(text.saveWithExtension(".html") );
 codeViewer.set_Text(text.htmlDecode().tidyHtml() , ".html");
 });

foreach(var cweEntry in xRoot.elements().remove(0))
{
 var entryData = cweEntry.elements();
 treeView.add_Node(entryData[0].value(),entryData[1].value());
}

treeView.selectFirst();
//O2File:API_VistaDB.cs
//O2File:HtmlAgilityPack_ExtensionMethods.cs

//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll

April 13, 2012 Posted by | CheckMarx | Leave a comment

VistaDB API and GUI (as used by Checkmarx SAST engine)

O2 now supports the VistaDB, which is a pure .NET embeded database used by applications like Checkmarx (who use it to support their SAST Web Application).

After using it a bit, I have to say that VistaDB seams like a really nice solution, since it is really a xcopy-based-relational-database with full SQL support.

To add support to VistaDB in O2, all that was needed was to grab the API_SqlServer.cs script and changed the Sql classes to their equivalent VistaDB classes (for example SqlConnection became VistaDBConnection).

Since VistaDB is not an open source project, you will need to have access to their eval download, or to an application that uses it (note how  used the VistaDB.NET20.dll from the Checkmarx install folder)

Util – VistaDB Browser.h2

//var topPanel = panel.clear().add_Panel();
var topPanel = O2Gui.open<Panel>("Sql Info",900,500);
topPanel.insert_Below(100).add_LogViewer();
var vistaDb = new API_VistaDB();
var show= false;
show = true;
TabControl tabControl = null;
Action loadDataFromCurrentConnection = 
    ()=>{
            tabControl.remove_Tab("Database details viewer");
            tabControl.remove_Tab("Table's Schema");
            tabControl.remove_Tab("Table's Data");
            tabControl.remove_Tab("Stored Procedures");
            vistaDb.add_Viewer_DataBases(tabControl.add_Tab("Database details viewer"));
            vistaDb.add_Viewer_Tables(tabControl.add_Tab("Table's Schema"));
            vistaDb.add_Viewer_TablesData(tabControl.add_Tab("Table's Data"));

        };



if(show)
{
    tabControl = topPanel.add_TabControl();    
    vistaDb.add_ConnectionStringTester(tabControl.add_Tab("Test/Set SqlConnection string"), loadDataFromCurrentConnection);
    vistaDb.add_GUI_SqlCommandExecute(tabControl.add_Tab("Execute SQL Commands"));                        
}



//O2Ref:System.Data.dll
//O2File:API_VistaDB.cs
//O2Tag_DontAddExtraO2Files

 
API_VistaDB.cs

using System;
using System.Data;
using System.Xml.Serialization;
using System.Linq;
using System.Drawing;
using System.Windows.Forms;
using System.Collections.Generic;
using O2.Interfaces.O2Core;
using O2.Kernel;
using O2.Kernel.ExtensionMethods;
using O2.DotNetWrappers.DotNet;
using O2.DotNetWrappers.ExtensionMethods;
using O2.External.SharpDevelop.ExtensionMethods;
using O2.Views.ASCX.ExtensionMethods;
using O2.XRules.Database.Utils;
using VistaDB.Provider;
//O2Ref:C:\Program Files\Checkmarx\Checkmarx Engine Server\VistaDB.NET20.dll</pre>
namespace O2.XRules.Database.APIs
{
    public class API_VistaDB_Test
    {
        public void launchTestGui()
        {
            "Util - VistaDB Browser.h2".local().executeH2Script();
        }
    }

    public class API_VistaDB
    {  
        public string ConnectionString { get;set; }
        public string LastError { get; set; }
       
        public API_VistaDB()
        {
            ConnectionString = @"data source='C:\Program Files\Checkmarx\Checkmarx Application Server\CxDB.vdb3'";     //default to this one           
        }       
       
        public API_VistaDB(string connectionString)
        {
            ConnectionString = connectionString;
        }
    }
   
    public class Database
    {
        public API_VistaDB VistaDB { get; set; }
        public string Name { get; set; }       
        public List<Table> Tables { get; set; }
        public List<StoredProcedure> StoredProcedures { get; set; }
       
        public Database(string name)
        {
            Name = name.trim();
            Tables = new List<Table>();
            StoredProcedures = new List<StoredProcedure>();
        }
       
        public Database(API_VistaDB vistaDB, string name) : this (name)
        {
            VistaDB = vistaDB;
       
        }
    }
   
    public class Table
    {
        [XmlIgnore] public API_VistaDB VistaDB { get; set; }               
//        public string Catalog {get;set;}
//        public string Schema {get;set;}
        public string Name {get;set;}
//        public string Type {get;set;}
        public List<Table_Column> Columns {get;set;}
       
        public DataTable TableData { get; set; }
       
        public Table()
        {
            Columns = new List<Table_Column>();
        }
       
        public override string ToString()
        {
            /*return (Schema.valid())
                        ? "{0}.{1}".format(Schema, Name)
                        : Name;*/
            return Name;           
        }
    }
   
    public class Table_Column
    {
        //name, typeId, objectId,options , scriptValue
        public string Name { get; set; }       
        public string TypeId { get; set; }       
        public string ObjectId { get; set; }       
        public string Options { get; set; }       
        public string ScriptValue     { get; set; }       
       
        public override string ToString()
        {
            //return "{0} ({1})".format(Name, DataType);
            return "{0} ({1})".format(Name);
        }
    }
   
    public class StoredProcedure
    {
        public string Schema {get;set;}
        public string Name {get;set;}
        public string Value {get;set;}
       
        public StoredProcedure(string schema, string name, string value)
        {
            Schema = schema;
            Name = name;
            Value = value;
        }
       
        public StoredProcedure(string name, string value) : this("",name, value)
        {
           
        }
       
        public override string ToString()
        {
            return (Schema.valid())
                        ? "{0}.{1}".format(Schema, Name)
                        : Name;
        }
    }
   
    //add these queries should be done using Linq
    public static class API_VistaDB_Helps
    {
        public static Database database(this API_VistaDB vistaDB, string name)
        {
            return new Database(vistaDB, name);
        }
    }
    public static class API_VistaDB_getData
    {
        public static List<string> database_Names(this API_VistaDB vistaDB)
        {
            var sqlQuery = "select * from [database schema] where typeid = 1";
            return (from DataRow row in vistaDB.executeReader(sqlQuery).Rows
                    select row["name"].str()).toList();
        }
       
        public static List<string> column_Names(this Table table)
        {
            return (from column in table.columns()
                       select column.Name).toList();
        }
       
        public static List<Table> tables(this API_VistaDB vistaDb)
        {           
            return vistaDb.database("").tables();
        }
       
        public static List<Table> tables(this Database database)
        {           
            if (database.Tables.size() ==0)
                database.map_Tables();
            return database.Tables;
        }
       
        public static Table table(this API_VistaDB vistaDb, string name)
        {
            return vistaDb.database("").table(name);
        }
       
        public static Table table(this Database database, string name)
        {           
            return (from table in database.tables()
                    where table.Name.trim() == name
                    select table).first();
        }
       
       
        public static List<Table_Column> columns(this Table table)
        {
            return table.Columns;
        }
    }
   
    public static class API_VistaDB_PopulateData
    {
        public static Database map_StoredProcedures(this Database database)
        {       
            var sqlQuery = "select Specific_Schema, Specific_Name, Routine_Definition  from {0}.Information_Schema.Routines".format(database.Name);
            var storedProceduresData = database.VistaDB.executeReader(sqlQuery);           
            foreach(DataRow row in storedProceduresData.Rows)
                database.StoredProcedures.Add(new StoredProcedure(row.ItemArray[0].str(),row.ItemArray[1].str(),row.ItemArray[2].str()));
            return database;
        }   
       
        public static Database map_Tables(this API_VistaDB vistaDB)
        {
            return vistaDB.database("").map_Tables();
        }
       
        public static Database map_Tables(this Database database)
        {       
            var sqlQuery = "select * from [database schema] where typeid = 1".format();
            var tables = database.VistaDB.executeReader(sqlQuery);           
            foreach(DataRow row in tables.Rows)
                database.Tables.Add(new Table(){
                                                    VistaDB = database.VistaDB,
//                                                    Catalog = row.ItemArray[0].str(),
//                                                    Schema = row.ItemArray[1].str(),
                                                    Name = row["name"].str().trim()
//                                                    Type = row.ItemArray[3].str()
                                                    });
            return database;
        }   
       
        public static Database map_Table_Columns(this Database database)
        {       
            foreach(var table in database.tables())
            {           
                //var sqlQuery = "select Column_Name, Column_Default, Is_Nullable, Data_Type, Character_Maximum_Length from {0}.Information_Schema.Columns where table_Schema='{1}' and table_name='{2}'"
                //                    .format(table.Catalog, table.Schema,table.Name);
                               
                var objectId = database.VistaDB.executeScalar("select objectId from [database schema] where typeid = 1 and name ='{0}' ".format(table.Name));
                var sqlQuery = "select name, typeId, objectId,options , scriptValue from [database schema] where foreignReference = '{0}' ".format(objectId);
                
                var columns = database.VistaDB.executeReader(sqlQuery);           
               
                foreach(DataRow row in columns.Rows)
                    table.Columns.Add(new Table_Column(){
                                                        Name =  row.ItemArray[0].str().trim(),
                                                        TypeId = row.ItemArray[1].str().trim(),
                                                        ObjectId = row.ItemArray[2].str().trim(),
                                                        Options = row.ItemArray[3].str().trim(),
                                                        ScriptValue = row.ItemArray[4].str().trim()
                                                        });
            }
            return database;
        }
       
        public static API_VistaDB map_Table_Data(this API_VistaDB vistaDB, Table table)
        {
//            var sqlQuery = "select * from [{0}].[{1}].[{2}]".format(table.Catalog,table.Schema, table.Name);               
            var sqlQuery = "select * from {0}".format(table.Name);                           
            table.TableData = vistaDB.executeReader(sqlQuery);
            return vistaDB;
        }
       
        public static Database map_Table_Data(this Database database, Table table)
        {
            database.VistaDB.map_Table_Data(table);
            return database;
        }
        public static Database map_Table_Data(this Database database)
        {
            "Mapping table data".info();
            var timer = new O2Timer("Mapped tabled data").start();
            foreach(var table in database.tables())                   
                database.map_Table_Data(table);
            timer.stop();
            return database;               
        }
       
        public static DataTable dataTable(this Table table)
        {
            if (table.isNull())
                return null;
            table.VistaDB.map_Table_Data(table);
            return table.TableData;
        }       
       
        public static string xml(this Table table)
        {           
            var dataSet = new DataSet();
            dataSet.Tables.Add(table.dataTable());
            return dataSet.GetXml();
        }
    }
   
    public static class API_VistaDB_Queries
    {
        public static VistaDBConnection getOpenConnection(this API_VistaDB vistaDB)
        {                       
            "[API_VistaDB] Opening Connection".info();
            try
            {
                var sqlConnection = new VistaDBConnection(vistaDB.ConnectionString);           
                sqlConnection.Open();
                return sqlConnection;
            }
            catch(Exception ex)
            {
                vistaDB.LastError = ex.Message;
                "[executeNonQuery] {0}".error(ex.Message);
                //ex.log();
            }           
            return null;
        }
       
        public static VistaDBConnection closeConnection(this API_VistaDB vistaDB, VistaDBConnection sqlConnection)
        {                       
            "[API_VistaDB] Closing Connection".info();
            try
            {               
                sqlConnection.Close();
                return sqlConnection;
            }
            catch(Exception ex)
            {
                vistaDB.LastError = ex.Message;
                "[executeNonQuery] {0}".error(ex.Message);
                //ex.log();
            }           
            return null;
        }
       
        public static API_VistaDB executeNonQuery(this API_VistaDB vistaDB, VistaDBConnection sqlConnection, string command)
        {           
            "[API_VistaDB] Executing Non Query: {0}".info(command);
            try
            {
                var sqlCommand = new VistaDBCommand();
                sqlCommand.Connection = sqlConnection;
                sqlCommand.CommandText = command;
                sqlCommand.CommandType = CommandType.Text;
                sqlCommand.ExecuteNonQuery();
            }
            catch(Exception ex)
            {
                vistaDB.LastError = ex.Message;
                "[executeNonQuery] {0}".error(ex.Message);
                //ex.log();
            }
            return vistaDB;
        }
       
        public static API_VistaDB executeNonQuery(this API_VistaDB vistaDB, string command)
        {       
            "[API_VistaDB] Executing Non Query: {0}".info(command);
            VistaDBConnection sqlConnection = null;
            try
            {
                sqlConnection = new VistaDBConnection(vistaDB.ConnectionString);           
                sqlConnection.Open();
                var sqlCommand = new VistaDBCommand();
                sqlCommand.Connection = sqlConnection;
                sqlCommand.CommandText = command;
                sqlCommand.CommandType = CommandType.Text;
                sqlCommand.ExecuteNonQuery();
            }
            catch(Exception ex)
            {
                vistaDB.LastError = ex.Message;
                "[executeNonQuery] {0}".error(ex.Message);
                //ex.log();
            }
            finally
            {
                if (sqlConnection.notNull())
                    sqlConnection.Close();
            }
            return vistaDB;
        }
       
        public static object executeScalar(this API_VistaDB vistaDB, string command)
        {   
            "[API_VistaDB] Executing Scalar: {0}".info(command);
            VistaDBConnection sqlConnection = null;
            try
            {
                sqlConnection = new VistaDBConnection(vistaDB.ConnectionString);
                sqlConnection.Open();
                var sqlCommand = new VistaDBCommand();
                sqlCommand.Connection = sqlConnection;
                sqlCommand.CommandText = command;
                sqlCommand.CommandType = CommandType.Text;
                return sqlCommand.ExecuteScalar();
            }
            catch(Exception ex)
            {
                vistaDB.LastError = ex.Message;
                "[executeNonQuery] {0}".error(ex.Message);
                //ex.log();
            }
            finally
            {
                sqlConnection.Close();
            }
            return null;
        }
       
        public static DataTable executeReader(this API_VistaDB vistaDB, string command)
        {
            var sqlConnection = new VistaDBConnection(vistaDB.ConnectionString);
            sqlConnection.Open();
            try
            {
                var sqlCommand = new VistaDBCommand();
                sqlCommand.Connection = sqlConnection;
                sqlCommand.CommandText = command;
                sqlCommand.CommandType = CommandType.Text;
                var reader =  sqlCommand.ExecuteReader();
                var dataTable = new DataTable();
                dataTable.Load(reader);
                return dataTable;
            }
            catch(Exception ex)
            {
                vistaDB.LastError = ex.Message;
                "[executeNonQuery] {0}".error(ex.Message);
                //ex.log();
            }
            finally
            {
                if (sqlConnection.notNull())
                    sqlConnection.Close();
            }
            return null;
        }
    }
       
    public static class API_VistaDB_GUI_Controls
    {
        public static T add_ConnectionStringTester<T>(this API_VistaDB vistaDB , T control, Action afterConnect)
            where T : Control
        {
            control.clear();
            var connectionString = control.add_GroupBox("Connection String").add_TextArea();
            var connectionStringSamples = connectionString.parent().insert_Left<Panel>(200).add_GroupBox("Sample Connection Strings")
                                                          .add_TreeView()
                                                          .afterSelect<string>((text)=> connectionString.set_Text(text));
            var connectPanel = connectionString.insert_Below<Panel>(200);
            var button = connectPanel.insert_Above<Panel>(25).add_Button("Connect").fill(); 
            var response = connectPanel.add_GroupBox("Response").add_TextArea();                       
           
            button.onClick(()=>{
                                    try
                                    {
                                        var text = connectionString.get_Text();
                                        vistaDB.ConnectionString = text;
                                        response.set_Text("Connecting using: {0}".format(text));
                                        var sqlConnection = new VistaDBConnection(text);
                                        sqlConnection.Open();
                                        response.set_Text("Connected ok");
                                        afterConnect();
                                    }
                                    catch(Exception ex)
                                    {
                                        vistaDB.LastError = ex.Message;
                                        response.set_Text("Error: {0}".format(ex.Message));
                                    }                       
                                   
                                });
           
            //connectionString.set_Text(@"Data Source=.\SQLExpress;Trusted_Connection=True");
            var sampleConnectionStrings = new List<string>();
            //from <a href="http://www.connectionstrings.com/sql-server-2005">http://www.connectionstrings.com/sql-server-2005</a>
            sampleConnectionStrings.add(@"data source='C:\Program Files\Checkmarx\Checkmarx Application Server\CxDB.vdb3'")
                                   .add(@"Data Source=.\SQLExpress;Trusted_Connection=True")
                                   .add(@"Data Source=myServerAddress;Initial Catalog=myDataBase;Integrated Security=SSPI")                                                  
                                   .add(@"Data Source=myServerAddress;Initial Catalog=myDataBase;User Id=myUsername;Password=myPassword;")
                                   .add(@"Data Source=190.190.200.100,1433;Network Library=DBMSSOCN;Initial Catalog=myDataBase;User ID=myUsername;Password=myPassword;")
                                   .add(@"Server=.\SQLExpress;AttachDbFilename=c:\mydbfile.mdf;Database=dbname; Trusted_Connection=Yes;")
                                   .add(@"Server=.\SQLExpress;AttachDbFilename=|DataDirectory|mydbfile.mdf; Database=dbname;Trusted_Connection=Yes;")
                                   .add(@"Data Source=.\SQLExpress;Integrated Security=true; AttachDbFilename=|DataDirectory|\mydb.mdf;User Instance=true;");
                                  
            connectionStringSamples.add_Nodes(sampleConnectionStrings).selectFirst();             
           
            button.click();
            return control;
           
        }
               
       
        public static API_VistaDB add_Viewer_QueryResult<T>(this API_VistaDB vistaDB , T control, string sqlQuery)
            where T : Control
        {    
            control.clear();
            var dataTable = vistaDB.executeReader(sqlQuery);            
            var dataGridView = control.add_DataGridView();
            dataGridView.DataError+= (sender,e) => { // " dataGridView error: {0}".error(e.Context);
                                                   };
            dataGridView.invokeOnThread(()=> dataGridView.DataSource = dataTable );           
            return vistaDB;
        }
       
        public static API_VistaDB add_Viewer_DataBases<T>(this API_VistaDB vistaDB , T control)
            where T : Control
        {
            var sqlQuery = "select * from [database schema] where typeid = 1";
            return vistaDB.add_Viewer_QueryResult(control, sqlQuery);
        }
       
        public static API_VistaDB add_Viewer_Tables_Raw<T>(this API_VistaDB vistaDB , T control, string databaseName)
            where T : Control
        {
            var objectId = vistaDB.executeScalar("select objectId from [database schema] where typeid = 1 and name ='{0}'".format(databaseName));
           
            var sqlQuery = "select * from [database schema] where typeid = 3 and foreignReference ='{0}'".format(objectId);
           
            return vistaDB.add_Viewer_QueryResult(control, sqlQuery);
        }
       
        public static API_VistaDB add_Viewer_StoredProcedures_Raw<T>(this API_VistaDB vistaDB , T control, string databaseName)
            where T : Control
        {
       
            var sqlQuery = "select * from {0}.Information_Schema.Routines".format(databaseName);
            return vistaDB.add_Viewer_QueryResult(control, sqlQuery);
        }
       
        public static API_VistaDB add_Viewer_StoredProcedures<T>(this API_VistaDB vistaDB , T control)
            where T : Control
        {
            control.clear();
            Database currentDatabase = null;
            var value = control.add_TextArea();   
            var storedProcedure_Names = value.insert_Left<Panel>(200).add_TreeView().sort();
            var database_Names = storedProcedure_Names.insert_Above<Panel>(100).add_TreeView().sort();
           
            var filter = storedProcedure_Names.insert_Above(20)
                                              .add_TextBox("Filter:","")
                                                .onTextChange((text)=>{
                                                                            storedProcedure_Names.clear();
                                                                            var result = (from storedProcedure in currentDatabase.StoredProcedures
                                                                                          where storedProcedure.Name.regEx(text)
                                                                                          select storedProcedure);
                                                                            storedProcedure_Names.add_Nodes(result);
                                                                        });
           
            database_Names.afterSelect<string>(
                (database_Name)=>{
                                    value.set_Text("");
                                    currentDatabase = new Database(vistaDB, database_Name);
                                    currentDatabase.map_StoredProcedures();                                   
                                    storedProcedure_Names.clear();                       
                                    storedProcedure_Names.add_Nodes(currentDatabase.StoredProcedures);
                                    storedProcedure_Names.selectFirst();
                                 });
           
            storedProcedure_Names.afterSelect<StoredProcedure>(
                (storedProcedure) => value.set_Text(storedProcedure.Value) );
           
            database_Names.add_Nodes(vistaDB.database_Names());
           
            database_Names.selectFirst();
            return vistaDB;
        }
       
       
        public static API_VistaDB add_Viewer_Tables<T>(this API_VistaDB vistaDB , T control)
            where T : Control
        {       
            control.clear();
            var value = control.add_TableList();   
            var tables_Names = value.insert_Left<Panel>(200).add_TreeView().sort();            
//            var database_Names = tables_Names.insert_Above<Panel>(100).add_TreeView().sort();
/*            database_Names.afterSelect<string>(
                (database_Name)=>{
                                    tables_Names.backColor(Color.Salmon);
                                    O2Thread.mtaThread(
                                        ()=>{
                                                value.set_Text("");
                                                var database = new Database(vistaDB, database_Name);                                   
                                                database.map_Tables()
                                                        .map_Table_Columns();
                                                tables_Names.clear();                       
                                                tables_Names.add_Nodes(database.Tables);
                                                tables_Names.selectFirst();
                                                tables_Names.backColor(Color.White);
                                            });    
                                 });
*/           
            tables_Names.afterSelect<Table>(
                (table) => value.show(table.Columns) );
           
//            database_Names.add_Nodes(vistaDB.database_Names());
           
//            database_Names.selectFirst();

            var database = new Database(vistaDB, "");                                   
            database.map_Tables()
                    .map_Table_Columns();
            tables_Names.clear();                       
            tables_Names.add_Nodes(database.Tables);
            tables_Names.selectFirst();
            tables_Names.backColor(Color.White);

            return vistaDB;
        }
       
        public static API_VistaDB add_Viewer_TablesData<T>(this API_VistaDB vistaDB , T control)
            where T : Control
        {       
            control.clear();
            var dataGridView = control.add_DataGridView();
           
            dataGridView.DataError+= (sender,e) => {}; //" dataGridView error: {0}".error(e.Context);};
            var tables_Names = dataGridView.insert_Left<Panel>(200).add_TreeView().sort();            
            var database_Names = tables_Names.insert_Above<Panel>(100).add_TreeView().sort();
            var preloadAllData = false;
            tables_Names.insert_Below(20).add_CheckBox("Preload all data from database",0,0,(value)=>preloadAllData = value).autoSize();//.check();
            var rowData = dataGridView.insert_Below<Panel>(100).add_SourceCodeViewer();
            var rowDataField = rowData.insert_Left<Panel>(100).add_TreeView();
            var selectedField = "";
           
            rowDataField.afterSelect<DataGridViewCell>(
                (cell)=>{
                            selectedField = rowDataField.selected().get_Text();
                            var fieldContent = cell.Value.str().fixCRLF();
                            if (fieldContent.starts("<?xml"))
                            {   
                                "mapping xml".info();
                                fieldContent = fieldContent.xmlFormat();
                                rowData.set_Text(fieldContent,"a.xml");
                            }
                            else
                                rowData.set_Text(fieldContent);
                        });
           
            dataGridView.afterSelect(
                (row)=> {                                                                                   
                            rowDataField.clear();
                            //rowData.set_Text("");
                            foreach(DataGridViewCell cell in row.Cells)
                            {
                                var fieldName = dataGridView.Columns[cell.ColumnIndex].Name;
                                var node = rowDataField.add_Node(fieldName,cell);
                                if (fieldName == selectedField)
                                    node.selected();
                            }
                            if (rowDataField.selected().isNull())
                                rowDataField.selectFirst();                                                                                       
                        });
                       
            database_Names.afterSelect<string>(
                (database_Name)=>{
                                    tables_Names.backColor(Color.Salmon);
                                    O2Thread.mtaThread(
                                        ()=>{
                                                var database = new Database(vistaDB, database_Name);                                   
                                                database.map_Tables();
                                                if (preloadAllData)                                                                                           
                                                    database.map_Table_Data();                                               
                                                tables_Names.clear();                       
                                                tables_Names.add_Nodes(database.Tables);
                                                tables_Names.selectFirst();
                                                tables_Names.backColor(Color.White);
                                               
                                                database_Names.splitContainer().panel1Collapsed(true);
                                            });
                                 });
           
            Action<Table> loadTableData =
                (table)=>{
                            tables_Names.backColor(Color.Salmon);
                            O2Thread.mtaThread(
                                        ()=>{
                                                rowDataField.clear();
                                                rowData.set_Text("");   
                                                dataGridView.remove_Columns();                           
                                                if (table.TableData.isNull())                           
                                                    vistaDB.map_Table_Data(table);                               
                                                dataGridView.invokeOnThread(()=>dataGridView.DataSource= table.TableData);       
                                                tables_Names.backColor(Color.White);
                                            });
                         };
            tables_Names.afterSelect<Table>(
                (table)=>{
                            loadTableData(table);
                         });
           
            database_Names.add_Nodes(vistaDB.database_Names());
           
            database_Names.selectFirst(); 
                       
           
            tables_Names.add_ContextMenu().add_MenuItem("reload data",
                ()=>{
                        var selectedNode = tables_Names.selected();
                        if (selectedNode.notNull())
                        {
                            var table = (Table)tables_Names.selected().get_Tag();
                            table.TableData = null;
                            loadTableData(table);
                        }
                    });
            return vistaDB;
        }
       
        public static API_VistaDB add_GUI_SqlCommandExecute<T>(this API_VistaDB vistaDB , T control)
            where T : Control
        {
            Action<string> executeNonQuery=null;
            Action<string> executeReader =null;
            var resultsPanel = control.add_GroupBox("Result") ; 
            var sqlCommandToExecute = resultsPanel.insert_Above("Sql Command to execute").add_TextArea();
            var sampleQueries = sqlCommandToExecute.insert_Left(300, "Sample Queries")
                                                   .add_TreeView()
                                                   .afterSelect<string>((text)=>sqlCommandToExecute.set_Text(text));
 
            sqlCommandToExecute.insert_Right(200)
                               .add_Button("Execute Non Query")
                               .fill()
                               .onClick(()=>{
                                                 "Executing Non Query".info();                                    
                                                 executeNonQuery(sqlCommandToExecute.get_Text());
                                             })
                              .insert_Above()
                              .add_Button("Execute Reader")
                              .fill()
                              .onClick(()=> {
                                                 "Executing Reader".info();
                                                 executeReader(sqlCommandToExecute.get_Text());
                                             });;
           
            executeReader = (sqlQuery)=>{
                                            vistaDB.add_Viewer_QueryResult(resultsPanel, sqlQuery);
                                            "done".info();
                                        };   
                                       
            executeNonQuery = (sqlText)=> {           
                                                var    log = resultsPanel.control<TextBox>();
                                                if (log.isNull())
                                                    log = resultsPanel.clear().add_TextArea();
                                                if (sqlText.contains("GO".line()))
                                                {                                       
                                                    var sqlTexts = sqlText.line().split("GO".line());
                                                    log.append_Line("[{0}]Found a GO, so breaking it into {1} queries".format(DateTime.Now,sqlTexts.size()));                                         
                                                    var sqlConnection = vistaDB.getOpenConnection();
                                                    foreach(var text in sqlTexts)                                                                                   
                                                    {               
                                                        vistaDB.executeNonQuery(sqlConnection, text);                                                                               
                                                       
                                                        if (vistaDB.LastError.valid())
                                                        {
                                                            log.append_Line("SQL ERROR: {0}".lineBeforeAndAfter().format(vistaDB.LastError));
                                                            log.append_Line("ERROR: stoping execution since there was an error which executing the query: {0}".format(text).lineBeforeAndAfter());
                                                            break;
                                                        }           
                                                    }
                                                    vistaDB.closeConnection(sqlConnection);
                                                }
                                                else
                                                    {
                                                        log.append_Line("Executing as Non Query: {0}".format(sqlText));
                                                        vistaDB.LastError = "";
                                                        vistaDB.executeNonQuery(sqlText);
                                                        if (vistaDB.LastError.valid())
                                                            log.append_Line("SQL ERROR: {0}".lineBeforeAndAfter().format(vistaDB.LastError));
                                                    }
                                                "done".info();
                                           };           
               
            sampleQueries.add_Nodes(new string[] {
                                                    "select * from master..sysDatabases",
                                                    "select * from master.Information_Schema.Tables",
                                                    "select * from master.Information_Schema.Routines"
                                                });
            sampleQueries.selectFirst();
            return vistaDB;
        }
    }       
}

 

April 13, 2012 Posted by | .NET, CheckMarx | 1 Comment

Quickly testing RegExes and “Util – Text RegEx using FuzzDb.h2” O2 script

If you want to quickly test a RegEx for possible security blind spots, here are a couple script samples that migth help you.

While writing these examples I ended up writing a mini tool which you can now access via the O2 Script Util – Text RegEx using FuzzDb.h2 and looks like this:

Source Code Snippets

Here are a couple script examples on how to test regexes with O2:

Here is a simple example:

var regExString = @"['""].*|[+\-*/%=&|^~'""]";

var payload1 = "this is ok";
var payload2 = "this is' ok";

var result1 = payload1.regEx(regExString);
var result2 = payload2.regEx(regExString);
return "{0} {1}".format(result1, result2);

Now add the fuzzDb payloads

var regExString = @"['""].*|[+\-*/%=&|^~'""]";

var matches = new List<string>();

var fuzzDb  = new API_FuzzDB();

foreach(var payload in fuzzDb.payloads_Xss())
    if (payload.regEx(regExString).isFalse())
        matches.add(payload);

foreach(var payload in fuzzDb.payloads_SQLi_Generic())
    if (payload.regEx(regExString).isFalse())
        matches.add(payload);
       
return matches;

//O2File:API_FuzzDB.cs

You can also test for what happens when the payloads are encoded


var regExString = @"['""].*|[+\-*/%=&|^~'""]";

var matches = new List<string>();

var fuzzDb  = new API_FuzzDB();

foreach(var payload in fuzzDb.payloads_Xss())
    if (payload.regEx(regExString.urlEncode()).isFalse())
        matches.add(payload.urlEncode());
       
return matches;
//O2File:API_FuzzDB.cs

Next step is to build a Gui:

var topPanel = panel.clear().add_Panel();

var actionsPanel = topPanel.insert_Above(40,"actions");
var dataGridView = topPanel.add_DataGridView()
                           .add_Columns("Payload", "Result");

var stop = false;
var sqli_payloads = false;
var xss_payloads = false;
var regExString = @"['""].*[+\-*/%=&|^~'""]";

Action startFuzzing =
    ()=>{
            var fuzzDb  = new API_FuzzDB();
            var startFuzzingLink = actionsPanel.link("Start Fuzzing").enabled(false);;
            var statusLabel = actionsPanel.controls<Label>(true).last();
           
            Action<List<string>> testPayloads =
                (payloads)=> {
                                foreach(var payload in payloads)
                                {               
                                    if (stop)
                                        break;
                                    statusLabel.set_Text("testing payload: {0}".format(payload));                                   
                                    if (payload.regEx(regExString).isFalse())               
                                        dataGridView.add_Row(payload, false);
                                }
                            };
            if (sqli_payloads)           
                testPayloads(fuzzDb.payloads_Xss());
            if (xss_payloads)           
                testPayloads(fuzzDb.payloads_Xss());   
               
            stop = false;
            startFuzzingLink.enabled(true);
            statusLabel.set_Text("Tests completed");
        };
       
actionsPanel.add_Label("RegEx To test").top(3)
            .append_TextBox(regExString).onTextChange((text)=> regExString = text).width(200)
            .append_CheckBox("Xss", (value)=> xss_payloads= value).tick().top(1)
            .append_CheckBox("Sqli", (value)=> sqli_payloads= value).tick()
            .append_Link("Start Fuzzing", ()=> startFuzzing()).font_bold().top(3)
            .append_Link("stop", ()=> stop = true)
            .append_Link("clear table", ()=> dataGridView.remove_Rows() )           
            .append_Label("...").autoSize().top(3);
           


startFuzzing();       
       
return "ok";
//O2File:API_FuzzDB.cs

Final version

Here is a version with a couple more features (see screenshot above)

var topPanel = "Util - Text RegEx using FuzzDb".popupWindow(1000,400);
//var topPanel = panel.clear().add_Panel();

var actionsPanel = topPanel.insert_Above(40,"actions");
var dataGridView = topPanel.add_DataGridView()
                           .add_Columns("Payload", "Result");

var stop = false;
var sqli_payloads = false;
var xss_payloads = false;
var withUrlEncoding = false;
var regExString = @"['""].*[+\-*/%=&|^~'""]";

Action startFuzzing =
    ()=>{
            var fuzzDb  = new API_FuzzDB();           
            var startFuzzingLink = actionsPanel.link("Start Fuzzing").enabled(false);;
            var statusLabel = actionsPanel.controls<Label>(true).last();
           
            Action<List<string>> testPayloads =
                (payloads)=> {
                                foreach(var payload in payloads)
                                {               
                                    if (stop)
                                        break;
                                    statusLabel.set_Text("testing payload: {0}".format(payload));                                   
                                    if (payload.regEx(regExString).isFalse())               
                                        dataGridView.add_Row(payload, false);
                                    if (withUrlEncoding)
                                    {
                                        var encodedPayload = payload.urlEncode();
                                        statusLabel.set_Text("testing payload: {0}".format(encodedPayload));                                   
                                        if (encodedPayload.regEx(regExString).isFalse())               
                                            dataGridView.add_Row(encodedPayload, false);
                                        this.sleep(100);   
                                    }
                                       
                                }
                            };
            if (sqli_payloads)           
                testPayloads(fuzzDb.payloads_SQLi_Generic());
            if (xss_payloads)           
                testPayloads(fuzzDb.payloads_Xss());   
            // we could also apply the transformation into the entire list like this
            //testPayloads( fuzzDb.payloads_Xss().Select((value)=> value.urlEncode())  );
            stop = false;
            startFuzzingLink.enabled(true);
            statusLabel.set_Text("Tests completed");
        };
       
actionsPanel.add_Label("RegEx To test").top(3)
            .append_TextBox(regExString).onTextChange((text)=> regExString = text).width(200)
            .append_CheckBox("Xss", (value)=> xss_payloads= value).tick().top(1)
            .append_CheckBox("Sqli", (value)=> sqli_payloads= value)//.tick()
            .append_CheckBox("with UrlEncoding", (value)=> withUrlEncoding= value)//.tick()
            .append_Link("Start Fuzzing", ()=> startFuzzing()).font_bold().top(3)
            .append_Link("stop", ()=> stop = true)
            .append_Link("clear table", ()=> dataGridView.remove_Rows() )           
            .append_Label("...").autoSize().top(3);
           


startFuzzing();       
       
return "ok";
//O2File:API_FuzzDB.cs

December 7, 2011 Posted by | FuzzDB, XSS | Leave a comment

API_ConsoleOut.cs – New and InProcess Capture of the Console Output

There are multiple scenarious where it is usefull to capture (via Lambda callbacks) the calls to Console.Write. Most solutions out there are based on using the Process class (via its ProcessStartInfo settings):

The problem with those solutions is they are able to capture the Console Output of the new process created, NOT the current process (which is what I want since I need to see the Console messages created by an external API/Code that I consuming from O2)

Using O2 to captute Console Output of new processes

Note: If all you want to do is to capture the console output of a new process, there are a bunch of O2 methods/APIs that can help you with it:

Which can be used like this:

run cmd.exe (with params: /c dir) and get its console output

return "cmd.exe".startProcess_getConsoleOut(" /c dir");

run netstat.exe (with params: -na) and redirect its console into a lambda method

"netstat.exe".startProcess("-na", (line)=> line.info());

run netstat.exe and show its results in a table list

var topPanel = panel.clear().add_Panel();
var tableList = topPanel.add_TableList();
var lines = "netstat.exe".startProcess_getConsoleOut("-na")
                         .lines()
                         .removeRange(0,2);
var netstatData = (from line in lines
                   let fragments = line.split(" ").removeEmpty()
                   select new {
                                   @Type = fragments.value(0) , 
                                   Local_Address = fragments.value(1) ,
                                   Foreign_Address = fragments.value(2) ,
                                   State = fragments.value(3)
                                  });                  
tableList.show(netstatData);

Using O2 to captute Console Output in the current Process

To capture the Console Output in the current process, I first tried to capture the usage of a MemoryStream in a way simular to how Console actually works (see O2 blog post Dynamically creating private/internal types and delegates (UserCallBack and AsyncStreamReader) ) , but a much easier way is to specify a TextWriter via Console.SetOut.

To make it easier to use, I created the API_ConsoleOut class which can be used like this:

Show Console Ouput in a new window

"Write this to the console :)".open_ConsoleOut();
Console.Write("another");
Console.WriteLine("test");
//O2File:API_ConsoleOut.cs

Which looks like this:

 Show Console Output in an existing panel/control

var topPanel = panel.clear().add_Panel();
topPanel.show_ConsoleOut(); //this will open as a child of an existing control
Console.Write("another ");
Console.WriteLine("test");
topPanel.insert_Right().add_SourceCodeViewer().set_Text("<h1>some html<h1>",".html");
Console.WriteLine("now inside a Windows Control");

return "done";
//O2File:API_ConsoleOut.cs

Which looks like this:

 

Test GUI with Console Input

If you want to test the Console Input and Output, there is also a test GUI that you can use

var apiConsoleOut = new API_ConsoleOut();
apiConsoleOut.show_ConsoleOut_TestGui();

Console.Write("another ");
Console.WriteLine("test");
Console.WriteLine("now with more options");

//O2File:API_ConsoleOut.cs

Which looks like this:

Source code of API_Console_Out.cs

using System;
using System.IO;
using System.Text;
using System.Drawing;
using System.Windows.Forms;
using System.Collections.Generic;
using O2.Interfaces.O2Core;
using O2.Kernel;
using O2.Kernel.ExtensionMethods;
using O2.DotNetWrappers.ExtensionMethods;
using O2.Views.ASCX.classes.MainGUI;
using O2.XRules.Database.Utils;

//O2File:_Extra_methods_Misc.cs
//O2File:_Extra_methods_Windows.cs
//O2File:_Extra_methods_WinForms_Controls.cs

namespace O2.XRules.Database.APIs
{
    public class API_ConsoleOut : Control
    {            
        public O2StreamWriter o2StreamWriter;
       
        public API_ConsoleOut()
        {       
            this.o2StreamWriter = new O2StreamWriter();     
            this.o2StreamWriter.Name = "API_ConsoleOut";
            Console.SetOut(this.o2StreamWriter); 
        }
    }
   
    public class O2StreamWriter : TextWriter
    {
        public StringBuilder ConsoleOut     { get; set; }
        public StringBuilder CurrentLine    { get; set; }
        public List<string> Lines            { get; set; }
        public Action<char> On_NewChar        { get; set; }   
        public Action<string> On_NewLine     { get; set; }
        public bool LogAllChars                { get; set; }
        public bool LogAllLines                { get; set; }
        public string Name                    { get; set; }
        public int PauseBetweenCharWrite    { get; set; }
       
        public O2StreamWriter()
        {
            ConsoleOut = new StringBuilder();
            CurrentLine = new StringBuilder();
            Lines = new List<string>();       
            LogAllChars = false;
            LogAllLines = true;
            Name = "O2StreamWriter";
        }
   
        public override void Write(char value)
        {
            base.Write(value);
              this.ConsoleOut.Append(value.ToString());             
             
              if (value == '\n')
                  handleLine();                 
              else
                  this.CurrentLine.Append(value.ToString());
                 
              if (this.LogAllChars)
                "[{0}] char: {1}".info(this.Name, value);
           
            if (this.On_NewChar.notNull())
                this.On_NewChar(value);
           
            if (PauseBetweenCharWrite > 0)
                this.sleep(PauseBetweenCharWrite, false);
        }               
       
        public void handleLine()
        {
            var line = this.CurrentLine.str();
            if (line.lastChar('\r'))
                line = line.removeLastChar();
            this.CurrentLine = new StringBuilder();
            this.Lines.add(line);
           
            if (this.LogAllLines)
                "[{0}]: {1}".info(this.Name, line);
               
            if (this.On_NewLine.notNull())
                this.On_NewLine(line);           
        }
        public override Encoding Encoding
        {
            get { return System.Text.Encoding.UTF8; }
        }
    }
   
    public static class API_ConsoleOut_ExtensionMethods_TestGuis
    {
       
        public static API_ConsoleOut show_ConsoleOut_TestGui(this API_ConsoleOut apiConsoleOut)
        {
            var topPanel = O2Gui.open<Panel>("Console.Out capture Test",700,200);
            return apiConsoleOut.show_ConsoleOut_TestGui(topPanel);
        }
       
       
        public static API_ConsoleOut show_ConsoleOut_TestGui(this API_ConsoleOut apiConsoleOut, Control topPanel)
        {
            var consoleIn = topPanel.insert_Above(40, "ConsoleIn");
            var lines_TextBox = topPanel.clear().add_GroupBox("Console.Out Lines").add_TreeView();
            var chars_TextBox = topPanel.insert_Right("Console.Out Chars").add_TextArea();
                       
            consoleIn.add_TextBox("Write to send to Console Input ->","")
                     .onKeyPress_getChar( (_char)=> { if (_char == '\x0d') Console.WriteLine(); else  Console.Write(_char); })
                     .multiLine(true)
                     .focus();
                                
            apiConsoleOut.o2StreamWriter.On_NewChar = (_char) => chars_TextBox.append_Text(_char.str());
            apiConsoleOut.o2StreamWriter.On_NewLine = (line) => lines_TextBox.add_Node(line);
           
            //apiConsoleOut.o2StreamWriter.PauseBetweenCharWrite = 1;            
           
            Console.WriteLine("Welcome to show_ConsoleOut_TestGui");   
            Console.Write("Type something on the TextBox above to see it here".line());
            return apiConsoleOut;
        }
       
        public static API_ConsoleOut open_ConsoleOut(this string message)
        {
            return message.show_ConsoleOut();
        }
       
        public static API_ConsoleOut show_ConsoleOut(this string message)
        {
            var apiConsoleOut = new API_ConsoleOut();
            apiConsoleOut.show_ConsoleOut();
            Console.WriteLine(message);
            return apiConsoleOut;
        }
       
        public static API_ConsoleOut add_ConsoleOut(this Control topPanel)
        {
            return topPanel.show_ConsoleOut();
        }
       
        public static API_ConsoleOut show_ConsoleOut<T>(this T topPanel)
            where T : Control
        {
            return topPanel.show_ConsoleOut("Welcome to O2's console");
        }
       
        public static API_ConsoleOut show_ConsoleOut<T>(this T topPanel, string message )
            where T : Control
        {
            var apiConsoleOut = new API_ConsoleOut();
            apiConsoleOut.show_ConsoleOut(topPanel);
            Console.WriteLine(message);
            return apiConsoleOut;
        }
       
        public static API_ConsoleOut show_ConsoleOut(this API_ConsoleOut apiConsoleOut)
        {
            var topPanel = O2Gui.open<Panel>("Console.Out",700,200);
            return apiConsoleOut.show_ConsoleOut(topPanel);
        }
       
       
        public static API_ConsoleOut show_ConsoleOut(this API_ConsoleOut apiConsoleOut, Control topPanel)
        {
            var consoleOut = topPanel.add_TextArea();
            consoleOut.backColor(Color.Black);
            consoleOut.foreColor(Color.White);
            consoleOut.font("Lucida Console");
            apiConsoleOut.o2StreamWriter.On_NewChar = (_char) => consoleOut.append_Text(_char.str());
            apiConsoleOut.o2StreamWriter.LogAllLines = false;
           
            consoleOut.add_ContextMenu()
                      .add_MenuItem("Clear", false, ()=> consoleOut.set_Text(""))
                      .add_MenuItem("Write to Console Input", ()=> Console.WriteLine("Text to send to Console.Input".askUser()));
            return apiConsoleOut;
        }
    }
}

November 26, 2011 Posted by | .NET, Tools | 3 Comments

O2 Script: Util – Util – Show Open Ports (via NetStat -afo).h2

Here is are two scripts that show how to parse ‘kind-off’ unstructured data into a table.

In the first example we grab the output from netstat -fa , parse it in order to extract the port numbers and display it on a table. The 2nd example uses netstat-fao which will also contain the process id (which is used by the script to find the process name)

For reference,  in netstat

  • a = Displays all connections and listening ports
  • f = Displays Fully Qualified Domain Names (FQDN) for foreign addresses.
  • o = Displays the owning process ID associated with each connection

1st version (netstat -fa)

var topPanel = O2Gui.open<Panel>("Util - Show Open Ports (via NetStat -na)",700,600);
//var topPanel = panel.clear().add_Panel();
var tableList = topPanel.add_TableList();

Action showNetStatOnTable =
    ()=>{
            var lines = "netstat.exe".startProcess_getConsoleOut("-na")
                                     .lines()
                                     .removeRange(0,2);
            var netstatData = (from line in lines
                               let fragments = line.split(" ").removeEmpty()
                               let value0 = fragments.value(0)
                               let value1 = fragments.value(1)
                               let value2 = fragments.value(2)
                               let value3 = fragments.value(3)
                               select new {
                                               @Type = value0 ,  
                                               //Local_Address = value1,
                                               Local_Address_IP = value1.replace("::","__").split(":")[0].replace("__","::"),
                                               Local_Address_Port =  value1.replace("::","__").split(":")[1] ,
                                               //Foreign_Address = value2 ,
                                               Foreign_Address_IP = value2.replace("::","__").split(":")[0].replace("__","::"),
                                               Foreign_Address_Port =  value2.replace("::","__").split(":")[1] ,
                                               State = value3
                                              });       
            tableList.show(netstatData);                                 
        }; 

topPanel.insert_Above(40, "Actions")
        .add_Link("Refresh", ()=> showNetStatOnTable())
        .click();

2nd version (netstat -fao)

var topPanel = O2Gui.open<Panel>("Util - Show Open Ports (via NetStat -afo)",700,600);
//var topPanel = panel.clear().add_Panel();

var tableList = topPanel.add_TableList();
tableList.title("Table showing parsed version of netstat command");
var selectedProcess = 0;

Action showNetStatOnTable =
    ()=>{
            var lines = "netstat.exe".startProcess_getConsoleOut("-afo")
                                     .lines()
                                     .removeRange(0,2);
            var netstatData = (from line in lines
                               let fragments = line.split(" ").removeEmpty()
                               let value0 = fragments.value(0)
                               let value1 = fragments.value(1)
                               let value2 = fragments.value(2)
                               let value3 = fragments.value(3)
                               let value4 = fragments.value(4)                              
                               select new {
                                               @Type = value0 ,  
                                               //Local_Address = value1,
                                               Local_Address_IP = value1.replace("::","__").split(":")[0].replace("__","::"),
                                               Local_Address_Port =  value1.replace("::","__").split(":")[1] ,
                                               //Foreign_Address = value2 ,
                                               Foreign_Address_IP = value2.replace("::","__").split(":")[0].replace("__","::"),
                                               Foreign_Address_Port =  value2.replace("::","__").split(":")[1] ,
                                               State = value3,
                                               Process_ID = value4,
                                               Process_Name = Processes.getProcess(value4.toInt()).ProcessName //,
                                               //Executable = value5
                                              });       
            tableList.show(netstatData);                                 
        }; 
       
Action stopSelectedProcess =
    ()=>{
            if (selectedProcess ==0)
                "Cannot Stop process with ID 0".error();
            else
            {
                var process = Processes.getProcess(selectedProcess);               
                "Stopping process: {0} (id: {1})".info(process.ProcessName, process.Id);
                process.stop()
                       .WaitForExit();
                showNetStatOnTable();
            }
        };

topPanel.insert_Above(40, "Actions")
        .add_Link("Refresh", ()=> showNetStatOnTable())
        .append_Link("Stop Selected Process", ()=> stopSelectedProcess()).leftAdd(100);
       

tableList.add_ContextMenu()
         .add_MenuItem("Stop Selected Process", ()=> stopSelectedProcess());
        
tableList.afterSelect_get_Cell(6,
            (value)=> {
                        selectedProcess = value.toInt();
                        "selectedProcess: {0}".info(selectedProcess);
                      });
                     
showNetStatOnTable();
//return "netstat.exe".startProcess_getConsoleOut("-afo");       
return "ok";       

This script is now part of O2’s Scripts as Util – Show Open Ports (via NetStat -afo).h2

November 26, 2011 Posted by | Network Security, Tools | Leave a comment

Consuming NMAP XML files and taking screenshots of web ports

Here is an O2 Script that consumes NMap saved XML files and takes screenshots of ports 80 and 443 ( both ip and hostnames)

The script is called Tool – Take Screenshots of NMap Web Ports.h2 and this is what it looks like when executed:

You can drag and drop NMap files into the left-most TreeView. This should trigger the file load and display the resolved URLS in that TreeView.

You can click on each entry on the Urls TreeView to take a screenshot for that page, or click on the ‘take screenshots of all pages’ link to process all links

Here is the result of dropping a nmap xml from a quick scan to owasp.org (the www.google.com entry was added for testing):

If you select an node from the Screenshots treeview, you will see the actual screenshot taken (note how in the picture above the browser is now showing a local file )

How was this script/tool created?

Here is how this script evolved:

start with a saved nmap results file:

var nmapSavedFile = @"o2platform.com.xml".local(); 

create xsd file (schema)

var targetDir = @"C:\O2\O2Scripts_Database\_Scripts\3rdParty_Tools\NMap";
var xsdFile = targetDir.pathCombine("nmap.xsd");
var nmapSavedFile = @"o2platform.com.xml".local();  
return nmapSavedFile.xmlCreateXSD().saveAs(xsdFile);

Since that worked, create CSharp file and copy to the correct location

var nmapSavedFile = @"o2platform.com.xml".local();  
var createdFile = nmapSavedFile.xmlCreateCSharpFile_Patched();
var targetDir = @"C:\O2\O2Scripts_Database\_Scripts\3rdParty_Tools\NMap";

var csFile = targetDir.pathCombine("nmap.cs");
Files.MoveFile(createdFile, csFile);
var xsdFile = targetDir.pathCombine("nmap.xsd");
Files.MoveFile(nmapSavedFile + ".xsd", xsdFile);Creaate

view created nmap.cs file in source code viewer:

var nmapSavedFile = @"nmap.cs".local();  
nmapSavedFile.showInCodeViewer();

load saved nmap xml file and view its contents:

var nmapSavedFile = @"o2platform.com.xml".local();  
var nmap = nmaprun.Load(nmapSavedFile);
nmap.details();
return nmap;

show results in table list

var topPanel = panel.clear().add_Panel();
var nmapSavedFile = @"o2platform.com.xml".local();  
var nmap = nmaprun.Load(nmapSavedFile);
var tableList = topPanel.add_TableList()
                        .add_Columns("Host", "Port", "Type", "State");
                       
                       
foreach(var host in nmap.host)
    foreach(var hostPort in host.ports)
        foreach(var port in hostPort.port)
            tableList.add_Row(host.address[0].addr.str(), 
                             port.portid.str(),
                             port.service[0].name.str(),
                             port.state[0].state.str());
tableList.setWidthToContent();

calculate the urls of the web ports

var webHosts = new List<string>();
foreach(var host in nmap.host)   
    foreach(var hostPort in host.ports)
        foreach(var port in hostPort.port)
            if (port.portid == "80" || port.portid == "443")
            {
                foreach(var address in host.address)       
                    webHosts.add("<a href="http://{0}:{1}&quot;.format(address.addr">http://{0}:{1}".format(address.addr</a>, port.portid));
                foreach(var hostHostname in host.hostnames)
                    foreach(var hostname in hostHostname.hostname)                           
                            webHosts.add("<a href="http://{0}:{1}&quot;.format(hostname.name">http://{0}:{1}".format(hostname.name</a>, port.portid));
            }
 
return webHosts;  

move web mapping into a Lambda method

Func<string, List<string>> resolveWebHosts =
    (nmapSavedFile) =>{
                        var nmap = nmaprun.Load(nmapSavedFile);
                        var webHosts = new List<string>();
                        foreach(var host in nmap.host)   
                            foreach(var hostPort in host.ports)
                                foreach(var port in hostPort.port)
                                    if (port.portid == "80" || port.portid == "443")
                                    {
                                        foreach(var address in host.address)       
                                            webHosts.add("<a href="http://{0}:{1}&quot;.format(address.addr">http://{0}:{1}".format(address.addr</a>, port.portid));
                                        foreach(var hostHostname in host.hostnames)
                                            foreach(var hostname in hostHostname.hostname)                           
                                                    webHosts.add("<a href="http://{0}:{1}&quot;.format(hostname.name">http://{0}:{1}".format(hostname.name</a>, port.portid));
                                    }
                        return webHosts;
                        };
 
var testFile = @"o2platform.com.xml".local();  
var urls = resolveWebHosts(testFile);
return urls;

showing urls in treeview, adding a WebBrowser control and opening selectect url in browser

//var topPanel = O2Gui.open<Panel>("{name}",700,400);
var topPanel = panel.clear().add_Panel();
Func<string, List<string>> resolveWebHosts =
    (nmapSavedFile) =>{
                        var nmap = nmaprun.Load(nmapSavedFile);                       
                        var webHosts = new List<string>();
                        foreach(var host in nmap.host)   
                            foreach(var hostPort in host.ports)
                                foreach(var port in hostPort.port)
                                    if (port.portid == "80" || port.portid == "443")
                                    {
                                        foreach(var address in host.address)       
                                            webHosts.add("<a href="http://{0}:{1}&quot;.format(address.addr">http://{0}:{1}".format(address.addr</a>, port.portid));
                                        foreach(var hostHostname in host.hostnames)
                                            foreach(var hostname in hostHostname.hostname)                           
                                                    webHosts.add("<a href="http://{0}:{1}&quot;.format(hostname.name">http://{0}:{1}".format(hostname.name</a>, port.portid));
                                    }
                        return webHosts;
                        };
 
var testFile = @"o2platform.com.xml".local();  
var urls = resolveWebHosts(testFile);

var treeView = topPanel.insert_Left(400,"Urls").add_TreeView();
var webBrowser = topPanel.add_WebBrowser_Control();
treeView.afterSelect<string>(
    (url)=> {                               
                webBrowser.open(url);               
            });

treeView.add_Nodes(urls)
        .selectFirst();
           
//O2File:nmap.cs
//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll

using O2’s IE/Watin Object instead

var testFile = @"o2platform.com.xml".local();  
var urls = resolveWebHosts(testFile);
var treeView = topPanel.insert_Left(400,"Urls").add_TreeView();
var ie = topPanel.add_IE();
treeView.afterSelect<string>(
    (url)=> {                               
                ie.open_ASync(url);                               
            });

treeView.add_Nodes(urls)
        .selectFirst();
           
//O2File:nmap.cs
//O2File:WatiN_IE_ExtensionMethods.cs
//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll
//O2Ref:WatiN.Core.1x.dll

Finally …

…here is the complete source code of this script

var topPanel = O2Gui.open<Panel>("Tool - Take Screenshots of NMap Web Ports",1200,400);
topPanel.insert_LogViewer();
//var topPanel = panel.clear().add_Panel();
var actionsPanel = topPanel.insert_Above(40,"Actions");
Func<string, List<string>> resolveWebHosts =
    (nmapSavedFile) =>{
                        var nmap = nmaprun.Load(nmapSavedFile);                       
                        var webHosts = new List<string>();
                        foreach(var host in nmap.host)   
                            foreach(var hostPort in host.ports)
                                foreach(var port in hostPort.port)
                                    if (port.portid == "80" || port.portid == "443")
                                    {
                                        var type = (port.portid == "80") ? "http" : "https";
                                        foreach(var address in host.address)       
                                            webHosts.add("{0}://{1}:{2}".format(type, address.addr, port.portid));
                                        foreach(var hostHostname in host.hostnames)
                                            foreach(var hostname in hostHostname.hostname)                            
                                                    webHosts.add("{0}://{1}:{2}".format(type,hostname.name, port.portid));
                                    }
                        return webHosts;
                        };
 

var targetFolder = "_nmapScreenshots".tempDir(false);
var urls = new List<string>();
var urls_TreeView = topPanel.insert_Left(400,"Urls (click to take screenshot)").add_TreeView();
var screenshots_TreeView = topPanel.insert_Left(400, "Screenshots").add_TreeView();
var ie = topPanel.add_IE_with_NavigationBar();//.silent(true);
var alertsHandler = ie.getAlertsHandler();     // auto closes popup-windows
var stopExecution = false;

Action<string> takeScreenshotOfWebPage =
    (url)=>{
                "taking screenshot of page: {0}".debug(url);
                ie.open(url);
                var screenshot = topPanel.screenshot();
                var screenshotFile     = targetFolder.pathCombine("{0}.jpg".format(url.safeFileName()));               
                screenshot.save(screenshotFile);
                screenshots_TreeView.add_Node(url, screenshotFile);
           };
Action takeScreenShotsOfAllPages =
    ()=>{
            screenshots_TreeView.clear();
            stopExecution = false;
            foreach(var url in urls)
                if(stopExecution.isFalse())
                    takeScreenshotOfWebPage(url);           
        };
       
screenshots_TreeView.afterSelect<string>(
    (bitmapFile)=>{
                    "here".info();
                    ie.open_ASync(bitmapFile);
                   });
                  
urls_TreeView.afterSelect<string>(
    (url)=> {
                screenshots_TreeView.pink();
                O2Thread.mtaThread(
                    ()=>{
                            takeScreenshotOfWebPage(url);
                            screenshots_TreeView.white();
                        });
            });

Action<string> loadNmapXmlFile=
    (file)=>{           
                urls_TreeView.clear();
                urls = resolveWebHosts(file);
                urls.add("<a href="http://www.google.com/">http://www.google.com</a>");
                urls_TreeView.add_Nodes(urls)
                              .selectFirst();
            };

urls_TreeView.onDrop(loadNmapXmlFile);       
actionsPanel.add_Link("take screnshots of all pages", takeScreenShotsOfAllPages)
            .append_Link("stop execution", ()=> stopExecution = true)
            .append_Link("View Folder with Screenshots", ()=> targetFolder.startProcess());

            
loadNmapXmlFile(@"o2platform.com.xml".local());    
return "ok";           
//O2File:nmap.cs
//O2File:API_Cropper.cs
//O2File:WatiN_IE_ExtensionMethods.cs
//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll
//O2Ref:WatiN.Core.1x.dll

Previous approach that is not working

Here are a couple scripts that document an aproach to load the nmap xml files using the DFT provided on nmap website site (which didn’t work)

download DTD from Nmap website

//var nmapDtd = "http://nmap.org/svn/docs/nmap.dtd".uri().download</a>();

Creating nmap.cs fil

var dtdFile = "nmap.xsd".local();
return dtdFile.xsdCreateCSharpFile();

Creating an empty nmap objec

var nmap = new nmaprun();
return nmap;
//using tempuri.org.nmap
//O2File:nmap.cs
//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll

loading saved nmap file

var nmapSavedFile = @"o2platform.com.xml".local(); 
var nmap = nmapSavedFile.load<nmaprun>();
return nmap;

show xml in sourcecode viewer and nmap object in property grid

var nmapSavedFile = @"o2platform.com.xml".local();  
var nmap = nmapSavedFile.load<nmaprun>();
topPanel.add_SourceCodeViewer()
        .set_Text(nmapSavedFile.fileContents().xmlFormat(),".xml");
topPanel.insert_Left(400,"loaded nmap file")
        .add_PropertyGrid()
        .show(nmap);

the problem with this approach is that it is not working as expected (the xml data is not being correctly loaded into the nmap object and nmaprun.Load(..file..) throws an exception)

November 16, 2011 Posted by | NMap, O2 Scripting, Tools | Leave a comment

Adding file comparison to O2 using (using WinMerge)

In order to quickly be able to compare files from an O2 script (using the great WinMerge tool), I’ve just added support for it.

You can now compare two files by simply making a call to the method file_Compare like this:

file1.file_Compare(file2);

That call will open up the WinMerge tool with those two files as parameters:

To enable this, there is a new API_WinMerge which can be used like this (the above screenshot is the result of this script execution):

var apiWinMerge = new API_WinMerge();
var file1 =  "WinMerge.cs".local(); 
var file2 =  "WinMerge.cs".local()
                          .fileContents()
                          .replace("WinMerge", "WineMerge_Changed!")
                          .save();

//apiWinMerge.openGui(); 
//apiWinMerge.compareTwoFiles(testFile1,testFile2);
file1.file_Compare(file2);
 
//O2File:WinMerge.cs

Note that the first time the API_WinMerge is used, WinMerge will be downloaded and  installed in the folder C:\O2\_tempDir\_ToolsOrAPIs\WinMerge\WinMerge-2.12.4-exe

Here is the source code of the API_WinMerge API (currently located at C:\O2\O2Scripts_Database\_Scripts\3rdParty_Tools\_Installers\WinMerge.cs)

using System;
using System.Diagnostics;
using O2.Kernel;
using O2.Kernel.ExtensionMethods;
using O2.DotNetWrappers.ExtensionMethods;
//O2File:Tool_API.cs
//O2File:_Extra_methods_To_Add_to_Main_CodeBase.cs

using O2.XRules.Database.Utils;
namespace O2.XRules.Database.APIs
{
    public class Install_WinMerge_Test
    {
        public void test()
        {
            new WinMerge().start();
        }
    }
   
    public class WinMerge : Tool_API
    {   
        public WinMerge() : this(true)
        {
        }
       
        public WinMerge(bool installNow)
        {
            config("WinMerge", "WinMerge v2.12.4", "WinMerge-2.12.4-exe.zip");           
            this.Install_Uri = "http://downloads.sourceforge.net/winmerge/WinMerge-2.12.4-exe.zip".uri();
            this.Executable = Install_Dir.pathCombine(@"WinMerge-2.12.4-exe\WinMergeU.exe");
            if (installNow)
                install();       
        }
       
       
        public bool install()
        {
            "Installing {0}".info(ToolName);
            return installFromZip_Web();                        
        }
       
        public Process start()
        {
            return start("");
        }
        public Process start(string parameters)
        {
            if (install())
                return this.Executable.startProcess(parameters);
            return null;
        }       
    }
   
    public class API_WinMerge
    {
        public WinMerge winMerge;
        public Process WinMergeProcess;
       
        public API_WinMerge()
        {
            winMerge = new WinMerge();
        }               
    }
   
    public static class API_WinMerge_ExtensionMethods
    {
        public static API_WinMerge openGui(this API_WinMerge apiWinMerge)
        {
            apiWinMerge.WinMergeProcess = apiWinMerge.winMerge.start();
            return apiWinMerge;
        }
       
        public static API_WinMerge compareTwoFiles(this API_WinMerge apiWinMerge, string file1, string file2)
        {
            var parameters = "\"{0}\" \"{1}".format(file1,file2);
            apiWinMerge.WinMergeProcess = apiWinMerge.winMerge.start(parameters);           
            return apiWinMerge;
        }
               
        public static API_WinMerge file_Compare(this string file1, string file2)
        {
            var winMerge = new API_WinMerge();
            return winMerge.compareTwoFiles(file1,file2);
        }
       
        public static API_WinMerge file_Merge(this string file1, string file2)
        {
            return file1.file_Compare(file2);
        }
    }
}

November 13, 2011 Posted by | Tools | Leave a comment

O2 Scripting Samples: Automating Rational AppScan Standard GUI and adding AppScan Standard Findings

Here are a number of smaller scripts I saved when working on the Showing IBM Rational AppScan Source Findings inside AppScan Standard (1st PoC) script. These are great if you are trying to get around O2’s APIs and get a feel for how a complex scripts ‘evolves’ 🙂

Note that these scripts have to be executed from inside the O2 Scriping environment created via Injecting O2 into IBM Rational AppScan Standard

 

Open up OpenScan dialog: 

var toolScriptButton = (ToolStripButton) MainForm.field("toolStripButtonOpen");
toolScriptButton.PerformClick();

show all values (public and private)  MainForm Fields and Properties

MainForm.details();

Get AppScan Debug Process Information

return AppScan.Diagnostics.Debug.GetProcessInformationLog();
//O2Ref:appscan.exe
//O2Ref:debug.dll

Get Logger

return AppScan.Diagnostics.Debug.Logger;
//O2Ref:appscan.exe
//O2Ref:debug.dll
//O2Ref:Utilities.dll

Get log file

var logger =  AppScan.Diagnostics.Debug.Logger;
var logFile = logger.LogFile;
return logFile;
//O2Ref:appscan.exe
//O2Ref:debug.dll
//O2Ref:Utilities.dll

View log file contents

var logger =  AppScan.Diagnostics.Debug.Logger;
var logFileContents = logger.LogFile.fileContents();
"AppScan Log file Contents".popupWindow()
                           .add_TextArea()
                           .set_Text(logFileContents);
//O2Ref:appscan.exe
//O2Ref:debug.dll
//O2Ref:Utilities.dll

View log file contents in a strongly typed table

var logger =  AppScan.Diagnostics.Debug.Logger;
var logFileContents = logger.LogFile.fileContents();


var logFileData =  (from  line in logFileContents.lines()       
        where line.split("|").size() > 1
       select line.split("|"))
        .Select((items)=>
           new {
                   process = items[0],
                   time = items[1],
                   message = items[2]
               } );
 
"AppScan Log file Contents".popupWindow()
                           .add_TableList()
                           .show(logFileData);                         

return "ok";
//O2Ref:appscan.exe
//O2Ref:debug.dll
//O2Ref:Utilities.dll

 
How to create the AppScan class (which is marked as internal)
Note that this doesn’t return a value (could be because it can only be called once). The constructor is correcly called but there in an internal AppScan error thrown   

var appScan =  "AppScanSDK.dll".assembly()
                               .type("AppScan")
                               .ctor();
                             
return appScan;
return "ok";
//using AppScan.GuiLayerImpl
//O2Ref:appscan.exe
//O2Ref:AppScanSDK.dll

Getting a object reference to an IAppScan object

return AppScan.AppScanFactory.CreateInstance();
//O2Ref:appscan.exe
//O2Ref:AppScanSDK.dll

View all (public & private) fields and properties from the IAppScan object (which is an object of type AppScan.GuiLayerImpl.AppScan)


var appScan =  AppScan.AppScanFactory.CreateInstance();
appScan.details();
//O2Ref:appscan.exe
//O2Ref:AppScanSDK.dll

Getting the scan object and returning it full type name (which is AppScan.GuiLayerImpl.Scan.ScanManager)

var appScan =  AppScan.AppScanFactory.CreateInstance();
var scan = appScan.Scan;
return scan.typeFullName();</pre>
//O2Ref:appscan.exe
//O2Ref:AppScanSDK.dll

 
Load scan (doesn’t work, hangs at 95%) 

var appScan =  AppScan.AppScanFactory.CreateInstance();
var scan = appScan.Scan;

var scanFileToLoad = @"C:\_WorkDir\IBM\Altoro\AltoroJ21.scan";
return scan.LoadScanData(scanFileToLoad);
 
//O2Ref:appscan.exe
//O2Ref:AppScanSDK.dll

 
Load scan ASync  (works)

var appScan =  AppScan.AppScanFactory.CreateInstance();
var scan = appScan.Scan;

var scanFileToLoad = @"C:\_WorkDir\IBM\Altoro\AltoroJ21.scan";
scan.LoadScanDataAsync(scanFileToLoad);
return "ok";

//O2Ref:appscan.exe
//O2Ref:AppScanSDK.dll

 
Get Reference to main TreeView with site tree, expand all nodes, and select the 5th

var scanTreeView = MainForm.controls<TreeView>(true)[0];
scanTreeView.expandAll();
var node = scanTreeView.allNodes()[5].selected();

return node;
//node.details();
//O2Ref:appscan.exe
//O2Ref:AppScanSDK.dll

 
Get the node type (which is AppScan.Gui.UserControls.TreeNodeWithPaint)

return node.typeFullName();

get the Full Path value (which is an AppScan Scan value )


return node.FullPath;</pre>

 
Show in O2’s LogViewer the FullPath value of the selected node 

var scanTreeView = MainForm.controls<TreeView>(true)[0];

scanTreeView.afterSelect<TreeNodeWithPaint>(
    (treeNodeWithPaint)=>
        {
            treeNodeWithPaint.FullPath.info();
        });
 
//using AppScan.Gui.UserControls
//O2Ref:appscan.exe
//O2Ref:UserControls.dll

Show source code of selected treenode in source code viewer (below the TreeView) 

var panelBelowTreeView = scanTreeView.splitContainer().Panel2;
panelBelowTreeView.clear();
var sourceCodeViewer = panelBelowTreeView.add_SourceCodeViewer();
var scanTreeView = MainForm.controls<TreeView>(true)[0];


var sourceFolder = @"C:\....\Altoro\AltoroJ2.1-src\WebContent";

var virtualFile = scanTreeView.selected().FullPath.remove(@"My Application\http://www.altoromutual.com:8080/\altoromutual\");
var pathToFile = sourceFolder.pathCombine(virtualFile); 
sourceCodeViewer.set_Text(pathToFile.fileContents(), ".aspx");
//using AppScan.Gui.UserControls
//O2Ref:appscan.exe
//O2Ref:UserControls.dll

Show source code on TreeNode Click


var sourceFolder = @"C:\...\Altoro\AltoroJ2.1-src\WebContent";

var scanTreeView = MainForm.controls<TreeView>(true)[0];

var panelBelowTreeView = scanTreeView.splitContainer().Panel2;
panelBelowTreeView.clear();
var sourceCodeViewer = panelBelowTreeView.add_SourceCodeViewer();

scanTreeView.afterSelect<TreeNodeWithPaint>(
    (treeNodeWithPaint)=>
        {
            try
            {           
            var virtualFile = scanTreeView.selected().FullPath.remove(@"My Application\http://www.altoromutual.com:8080/\altoromutual\");
            var pathToFile = sourceFolder.pathCombine(virtualFile); 
            if (pathToFile.fileExists())
            {
                sourceCodeViewer.open(pathToFile);
                if (pathToFile.extension(".jsp"))
                    sourceCodeViewer.editor().setDocumentHighlightingStrategy(".aspx");
            }
                else
                    sourceCodeViewer.set_Text("");
        }
        catch()
        {}
        });

return "ok";
//using AppScan.Gui.UserControls
//O2Ref:appscan.exe
//O2Ref:UserControls.dll

Add a top menu (to AppScan) with an menu item to open an *.scan file

Action<string> openScanFile =
    (scanFileToLoad)=>{
                            if (scanFileToLoad.fileExists())
                            {
                                var appScan =  AppScan.AppScanFactory.CreateInstance();
                                appScan.Scan.LoadScanDataAsync(scanFileToLoad);
                            }
                           
                        };

 
var menuStrip =  MainForm.controls<MenuStrip>(); 
var o2Menu = menuStrip.add_MenuItem("AltoroJ"); 
o2Menu.add_MenuItem("Open AltoroJ", ()=>  openScanFile("Where is the AltoroJ *.Scan file".askUser()) );

adding two menu items (one to load the scan file and another to configure the source code mapings)

Action<string> openScanFile =
    (scanFileToLoad)=>{
                            if (scanFileToLoad.fileExists())
                            {
                                var appScan =  AppScan.AppScanFactory.CreateInstance();
                                appScan.Scan.LoadScanDataAsync(scanFileToLoad);
                            }
                           
                        };

//run this only once
Action<string> configureTreeView =
    (sourceFolder)
        =>{
            if (sourceFolder.dirExists().isFalse())
                return;

            var scanTreeView = MainForm.controls<TreeView>(true)[0];
           
            var panelBelowTreeView = scanTreeView.splitContainer().Panel2;
            panelBelowTreeView.clear();
            var sourceCodeViewer = panelBelowTreeView.add_SourceCodeViewer();
           
            scanTreeView.afterSelect<TreeNodeWithPaint>(
                (treeNodeWithPaint)=>
                    {
                        try
                        {           
                            var virtualFile = scanTreeView.selected().FullPath.remove(@"My Application\http://www.altoromutual.com:8080/\altoromutual\");
                            var pathToFile = sourceFolder.pathCombine(virtualFile); 
                            if (pathToFile.fileExists())
                            {
                                sourceCodeViewer.open(pathToFile);
                                if (pathToFile.extension(".jsp"))
                                    sourceCodeViewer.editor().setDocumentHighlightingStrategy(".aspx");
                            }
                                else
                                    sourceCodeViewer.set_Text("");
                        }
                        catch
                        {
                        }
                    });

        };
 
var menuStrip =  MainForm.controls<MenuStrip>(); 
var o2Menu = menuStrip.add_MenuItem("AltoroJ"); 
o2Menu.add_MenuItem("Open AltoroJ", ()=>  openScanFile("Where is the AltoroJ *.Scan file".askUser()) );
o2Menu.add_MenuItem("Configure AltoroJ source code mapping", ()=>  configureTreeView("Where is the AltoroJ source code".askUser()) );

return "ok";

//using AppScan.Gui.UserControls
//O2Ref:appscan.exe
//O2Ref:UserControls.dll
//O2Ref:AppScanSDK.dll

Next step is to add the source code viewer for the AltoroJ AS.Source findings. Lets start by finding the multiple TabControls

return MainForm.controls<TabControl>(true);
//O2Ref:appscan.exe

Adding a test tab to the selected finding TabControl

var selectedFindingTabControl =  MainForm.controls<TabControl>(true)[1];
selectedFindingTabControl.add_Tab("Test");
//O2Ref:appscan.exe

Adding a findings Viewer

var selectedFindingTabControl =  MainForm.controls<TabControl>(true)[1];
var findingsTab = selectedFindingTabControl.add_Tab("SAST Findings");
var findingsViewer = findingsTab.add_FindingsViewer();

//O2Ref:appscan.exe
//using O2.XRules.Database.Findings
//O2File:Findings_ExtensionMethods.cs

load the findings  data and show it

            var o2Findings = ozasmtFile.loadO2Findings();

            var selectedFindingTabControl =  MainForm.controls<TabControl>(true)[1];
            //selectedFindingTabControl.TabPages.Clear();

            var allFindingsViewer =  selectedFindingTabControl.add_Tab("All SAST Findings ")
                                                  .add_FindingsViewer(false);
            var findingsMappedToFileViewer = selectedFindingTabControl.add_Tab("SAST Findings Mapped to Selecteded File")
                                                          .add_FindingsViewer(true);
            allFindingsViewer.show(o2Findings);

            Action<string> showFilteredResults =
                (filter)=>{   
                              var filteredResults = (from o2Finding in o2Findings
                                                     where o2Finding.file.contains(filter)
                                                     select o2Finding).toList();
                              findingsMappedToFileViewer.show(filteredResults);                 
                         };
           

            var scanTreeView = MainForm.controls<TreeView>(true)[0];                     

                scanTreeView.afterSelect<TreeNodeWithPaint>(
                    (treeNodeWithPaint)=>
                        {
                            try
                            {           
                                var virtualFile = scanTreeView.selected().FullPath.remove(@"My Application\http://www.altoromutual.com:8080/\altoromutual\");
                                //var pathToFile = sourceFolder.pathCombine(virtualFile); 
                                showFilteredResults(virtualFile);
                            }
                            catch {}
                        });                   

November 8, 2011 Posted by | .NET, AppScan | Leave a comment

Showing IBM Rational AppScan Source Findings inside AppScan Standard (1st PoC)

Here is a PoC I wrote last night for an IBMer which shows a very first rough pass at loading up IBM Rational AppScan Source findings inside AppScan Standard. The demo uses AltoroJ and at the moment connects the *.jsp AppScan Standard findings into the respective AppScan Source findings.

This is what it looks like after the O2 script has been executed and the data loaded:

for reference this is what AppScan Standard looks like without  the O2 Script execution (and the injection of the AppScan Source findings:


How to replicate this?

At the moment there are a couple steps that need to be done in order to get there:

1) start AppScan under O2 control by executing the Util – Launch AppScan Standard (O2 Version).h2 script (see Injecting O2 into IBM Rational AppScan Standard for details on how this works)

2) After AppScan loads (and the O2 .NET injection occours), you should see an O2 menu (on top) and an O2 Scripting environment on the middle left:

3) now drag and drop the Inside AppScan Standard- Add AltoroJ Menu.h2 script into the O2 Scripting environment, and execute it (note the addition of a new AltoroJ Menu after execution)

4) the AltoroJ has 3 options:

 – Open AppScan Standard File,  
 – Configure Source code and
 – Map AppScan Source findings

(the sub menus text is a but different, but that’s what it means)

Note that these need to opened in sequence

5) select ‘Open AltoroJ’ and enter the path to the *.scan file:

this will trigger the load of the saved assessment and should look like this when completed:

6) Next click on ‘Configure AltoroJ source code Mapping’ and enter the path to the source code (note that in this version of the script you need to point to the WebContent folder)

7) test the source code mappings by clicking on a*.jsp  file that exists on the file system

Note how the O2 Scripting control (middle left) was replaced with a SourceCodeViewer control that shows the content of the selected file (in the treeview above it).

Here is an example of another *.jsp file (note that the other servlet mappings are currently not supported (but it is not hard to do since we have access to all artifacts we need))

8) the final step is to select the menu option ‘Map AltoroJ SAST findings’ and enter the location of the AppScan Source saved findings file

9) View all Source code Findings

The last script loaded the findings in memory and added two Tabs to the bottom right tab Control.

The first tab added is called ‘All SAST Findings’ and will show details of all findings loaded

Including a view to see the Finding’s Traces:

10) View the findings that are mapped to the *.Jsp file currently selected

The other tab that was added to the TabControl is called ‘SAST Findings Mapped to Selected File’ and will only show the findings that exist for the file currently loaded:

Here is another example: (note how powerfull it is to see the BlackBox findings on the top and the Source Code findings on the bottom)

11) Next Steps?

There is still a LOT that needs to be done before this script is a good representation of the information that we already have available from both scans

For example one will need to consume the web.xml file in order to get the bean/url mappings from it (we also need to add proper color coding for the *.jsp file loaded on the right , like what happens on the left).  

Another interresting option is to look at the WAFL (Web Application Framework Language) file created by AppScan Source and see if we can extract from it the mappings between the Urls and its the Controllers

November 8, 2011 Posted by | AppScan, Java, Tools | 1 Comment