Extrating OWASP Chapter data via MediaWiki API and viewing it
Here is a script that uses O2’s MediaWiki API to extract OWASP’s chapter information from its WIKI.
Once the data is collected, a mini gui is used to show it:
var topPanel = panel.clear().add_Panel();
var owaspWiki = new OwaspWikiAPI(false);
var treeView = topPanel.add_TreeView();
var textArea = topPanel.insert_Right().add_TextArea();
var webBrowser = textArea.insert_Below().add_WebBrowser_Control();
Action<List<string>,TreeNode, bool> loadArea = null;
var addedCategories = new List<string>();
treeView.afterSelect<string>(
(page)=>{
textArea.set_Text(page);
var wikiText = owaspWiki.wikiText(page);
var htmlCode = owaspWiki.html(page);
textArea.set_Text(wikiText);
webBrowser.open(htmlCode.save());
});
loadArea = (pages, parentNode, addChapters) =>
{
foreach(var page in pages)
{
if (page.contains("Category") && addedCategories.contains(page).isFalse())
{
addedCategories.add(page);
var categoryNode = parentNode.add_Node(page);
var chaptersInArea = owaspWiki.pagesInCategory(page);
loadArea(chaptersInArea, categoryNode , true);
}
else
if (addChapters)
parentNode.add_Node(page);
}
};
var chapters = owaspWiki.pagesInCategory("Category:OWASP Chapter");
//var chapters = @"C:\O2\_tempDir\2-8-2012\tmpA92F.tmp.xml".load<List<string>>();
loadArea(chapters,treeView.rootNode(),false);
treeView.selectFirst();
//return pages.size();
return "ok";
//O2File:OwaspWikiAPI.cs
//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll
Example of creating, consuming and serializing C# classes
I’m currently helping Sarah Baso (from OWASP) to use O2 to streamline a number of OWASP admin tasks, and here are a couple script samples from the code that she is writing on retrieving details from Mailman Mailing lists
Here is the base class with data
// This file is part of the OWASP O2 Platform (http://www.owasp.org/index.php/OWASP_O2_Platform) and is released under the Apache 2.0 License (<a href="http://www.apache.org/licenses/LICENSE-2.0">http://www.apache.org/licenses/LICENSE-2.0</a>)
using System;
using System.Linq;
using System.Collections.Generic;
using System.Windows.Forms;
using System.Text;
using O2.Kernel;
using O2.Kernel.ExtensionMethods;
using O2.DotNetWrappers.DotNet;
using O2.DotNetWrappers.Windows;
using O2.DotNetWrappers.ExtensionMethods;
using O2.Views.ASCX.classes.MainGUI;
using O2.Views.ASCX.ExtensionMethods;
namespace O2.Script
{
public class MailingLists
{
public DateTime DateCollected { get; set;}
public List<MailingList> Lists { get; set; }
public MailingLists()
{
Lists = new List<MailingList>();
}
}
public class MailingList
{
public string Name { get; set; }
public bool Public { get; set; }
public List<Subscriber> Admins { get; set; }
public List<Subscriber> Subscribers { get; set; }
public MailingList()
{
Admins = new List<Subscriber>();
Subscribers = new List<Subscriber>();
}
}
public class Subscriber
{
public string Name { get; set; }
public string Email { get; set; }
}
}
Here is how that data can be consumed
var mailingLists = new MailingLists();</pre>
Action<string, bool> addList =
(listName, isPublic)=>
{
var mailingList = new MailingList();
mailingList.Name = listName;
mailingList.Public = isPublic;
mailingLists.Lists.Add(mailingList);
};
addList("owasp-london", true);
addList("owasp-rh", true);
addList("owasp-chapter", true);
/*var londonList = new MailingList();
londonList.Name = "Owasp-london";
londonList.Public = true;
mailingLists.Lists.Add(londonList);
var chaptersList = new MailingList();
chaptersList.Name = "Owasp-chapter";
chaptersList.Public = true;
mailingLists.Lists.Add(chaptersList);
*/
//return mailingLists;
//return myData;
var result = mailingLists.serialize(false);
panel.clear()
.add_TextArea()
.set_Text(result)
.size(16);
return mailingLists;
//using O2.Script
//O2File:C:\O2\_XRules_Local\Sarah's Test\MyData.cs
Here is how to normalize the email addresses
var url = https://lists.owasp.org/mailman/options/owasp-xyz-list/aaa_at_gmail.com;
if (url.contains("https://lists.owasp.org/mailman/options/"))
{
var items = url.remove("https://lists.owasp.org/mailman/options/").split("/");
var listName = items[0];
var email = items[1].replace("_at_","@");
return email;
}
return "no data";
OWASP MediaWiki Tool – File Uploader (O2 Script)
Following a request from a fellow OWASP Leader for an easier way to upload files to the OWASP wiki, here is an O2 script that allows the uploading of file(s) by just drag-and-droping then from a local folder (note that not all file types are currenly supported by the OWASP Wiki).
The script is called “Tool – OWASP Wiki – File Uploader.h2” and here is what the GUI looks like:
This script supports the upload of all files in a folder, but be careful when using it, since (in this version) there is no stop botton, and if you make a mistake, you might end uploading the wrong files to the OWASP wiki 🙂
The script was created using the very powerful O2 MediaWiki APIs, who can also be seen in action in these two YouTube videos:
- O2 Platform – MediaWiki Page Editor – Viewing, Editing and Creating content
- O2 Platform – MediaWiki Editor – Create Backup
Here is this script source code:
var userAccount = @"accounts.xml".credential("owasp.dinis");
var permitedFileTypes = "vsd, odp, gif, png, jpg, jpeg, doc, ppt, mp3, pdf, psd, zip, tar, tar.gz, tar.bz2, jar, docx, pptx, xls, xlsx";
//var topPanel = panel.clear().add_Panel();
var topPanel = "Tool - OWASP Wiki - File Uploader : {0}".format(permitedFileTypes).popupWindow(1000,500);
topPanel.insert_LogViewer();
var actionsPanel = topPanel.insert_Above(40,"Actions");
var recentChanges = topPanel.insert_Right("Recent Changes to OWASP Wiki");
var filesToUpload = topPanel.insert_Left(recentChanges.width()/2, "Files To Upload").add_TreeView();
var filesUploaded = topPanel.add_GroupBox("Files Uploaded").add_TextArea().wordWrap(false);
Label messageLabel = null;
var wikiApi = new OwaspWikiAPI(false);
Action showRecentChanges = `
()=>{
wikiApi.add_Table_with_RecentChanges(recentChanges.clear());
};
Action login =
()=>{
if (userAccount.isNull())
userAccount = ascx_AskUserForLoginDetails.ask();
if (userAccount.isNull() || userAccount.UserName.inValid() )
messageLabel.set_Text("Error, no user provided ").textColor(Color.Red);
else
{
messageLabel.set_Text("Logging in as user: {0}".format(userAccount.UserName)).textColor(Color.DarkOrange);
if (wikiApi.login(userAccount.UserName, userAccount.Password))
{
messageLabel.set_Text("Logged in OK as: {0}".format(userAccount.UserName)).textColor(Color.Green);
filesToUpload.enabled(true);
filesUploaded.enabled(true);
//showRecentChanges();
return;
}
messageLabel.set_Text("Failed to Login as : {0}".format(userAccount.UserName)).textColor(Color.Red);
}
userAccount = null;
};
Action<string> uploadFile =
(fileToUpload)=>{
filesToUpload.add_Node(fileToUpload);
filesToUpload.pink();
O2Thread.mtaThread(
()=>{
var fileUrl = wikiApi.uploadImage(fileToUpload);
if (fileUrl.valid())
{
filesUploaded.append_Line("Uploaded OK: {0} ->".line() +
"\t{1}", fileToUpload , fileUrl);
messageLabel.set_Text("Uploaded file: {0}".format(fileToUpload)).textColor(Color.Black);
showRecentChanges();
}
else
filesUploaded.append_Line("Error uploading: {0}", fileToUpload);
filesToUpload.white();
});
};
messageLabel = actionsPanel.add_Link("refresh recent changes table", ()=>showRecentChanges())
.append_Link("Login", ()=> login())
.append_Label("").autoSize();
filesToUpload.enabled(false);
filesUploaded.enabled(false);
messageLabel.set_Text("To Upload files, please login");
filesToUpload.add_Node("After login, drop file here to upload");
filesToUpload.onDrop(
(fileOrDir)=>
{
if(fileOrDir.dirExists())
foreach(var file in fileOrDir.files())
uploadFile(file);
else
uploadFile(fileOrDir);
});
login();
showRecentChanges();
//O2File:OwaspWikiAPI.cs
//O2File:ISecretData.cs
//O2File:ascx_AskUserForLoginDetails
Mapping out the OWASP Mailman admins (from web exposed data)
There was a recent thread on the OWASP-leaders list about unsubscribing to that list. The problem was caused by the auto population of that list with the current admins of an owasp mailing list.
As you can see on https://lists.owasp.org/mailman/admin , the problem is that there are a LOT of OWASP mailing lists (466 to be specific) and it is not easy to find out who is the current admin of what list.
To help with these mappings, I just wrote a quick O2 script which allows the collection and visualization of that data.
The script is called OWASP Mailing Lists – admins mappings.h2 and this is what it looks like when executed:
And here is its source code:
var hideTableDuringLoad = true;
var topPanel = "OWASP Mailing Lists - List admin mappings/emails".popupWindow(1200,500);
//var topPanel = panel.clear().add_Panel();
var tableList = topPanel.add_GroupBox("OWASP Mailing list mappigs").add_TableList();
var browser = topPanel.insert_Right("List Admin WebPage").add_WebBrowser_Control();
tableList.afterSelect_get_Cell(3,
(href)=>{
browser.open(href);
});
tableList.add_Columns("#", "email", "list", "href" );
var tempDir = "_owasp-leaders_mainmanMappings".tempDir(false);
var baseUrl = "https://lists.owasp.org/mailman/admin";
Func<string, string> getHtml_UsingCache =
(urlToGet)=> {
var cacheFilePath = tempDir.pathCombine(urlToGet.safeFileName() + ".html");
return (cacheFilePath.fileExists())
? cacheFilePath.fileContents()
: urlToGet.uri()
.getHtml()
.saveAs(cacheFilePath)
.fileContents();
};
Func<string,string, List<HtmlNode>> getNodesFromHtmlPage =
(urlToGet, filter)=> {
var htmlDocument = getHtml_UsingCache(urlToGet).htmlDocument();
return htmlDocument.select(filter);
};
//return getNodesFromHtmlPage(baseUrl, "//a");
var linksToFollow = (from link in getNodesFromHtmlPage(baseUrl, "//a")
where link.attribute("href").value().contains("mailman/admin")
select link).toList();
tableList.visible(false);
var listNumber = 1;
foreach(var linkToFollow in linksToFollow)
{
var href = linkToFollow.attribute("href").value();
var links = getNodesFromHtmlPage(href, "//a");
foreach(var link in links)
if (link.InnerText.contains(" at "))
foreach(var email in link.InnerText.split(","))
tableList.add_Row(
email.replace(" at <a href="mailto:%22,%20%22@%22).trim().ToLower">", "@").trim().ToLower</a>(),
linkToFollow.InnerText,
listNumber++.str() ,
href);
}
tableList.setWidthToContent();
tableList.visible(true);
tableList.selectFirst();
return "done";
//using HtmlAgilityPack
//using O2.XRules.Database.Utils.ExtensionMethods
//O2File:HtmlAgilityPack_ExtensionMethods.cs
//O2Ref:O2_Misc_Microsoft_MPL_Libs.dll
About
The O2 platform represents a new paradigm for how to perform, document and distribute Web Application security reviews. O2 is designed to Automate Security Consultants Knowledge and Workflows and to Allow non-security experts to access and consume Security Knowledge.
O2 can also be a very powerful prototyping and fast-development tool for .NET. Most O2 APIs are written using a Fluent API design, and its core has been published as a separate project called FluentSharp (hosted at CodePlex)
Download
Windows ClickOnce Installer:
Mailing List
The best place to keep updated with the lastest news and developers it to subscribe to the OWASP O2 Platform Mailing list (you can read its archives here)
OWASP O2 Platform Blog 