How to crawl websites in multiple languages using C#

In this article we will create a console app to crawl a webpage and read data using c#. We can use HttpWebRequest and HttpWebResponse also to get response from a webpage but I used httpagilitypack to read data in the example given below. It is much more easier to read the title, header and other html tags using htmlagility.

1)      Create a console application.

2)      Download htmlagilitypack from http://htmlagilitypack.codeplex.com/

3)      Copy htmlagility pack DLL to folder inside the project and add reference to HtmlAgilityPack.dll

4)      Next create a webclient and pass the URL to read to webclient. To read content in all languages (like Chinese, Japanese, Indonesian, Russian etc) make sure that the encoding is UTF8.

Complete code for crawling and reading data from webpage is given below

class Program

    {

        static void Main(string[] args)

        {

            WebClient webclient = new WebClient();

            HtmlDocument htmlDoc = new HtmlDocument();

            htmlDoc.Load(webclient.OpenRead("http://deebujacob.blogspot.com/2013/03/rendering-multiple-series-in-highcharts.html"), Encoding.UTF8);

            Console.WriteLine(htmlDoc.DocumentNode.SelectSingleNode("//title").InnerText);

            Console.WriteLine(htmlDoc.DocumentNode.SelectSingleNode("//body//h1").InnerText);

            Console.ReadLine();

        }

    }

How to store different languages in SQL Server

To store different languages (like Chinese, Japanese etc)  in SQL server the data type of the column should be NVARCHAR.

Also make sure that the stored procedure used to store data also has NVARCHAR parameter. If you are using SQL statement to directly insert data into the table then use "N" before the data.

Example given below

Create table #temp(data nvarchar(100))

INSERT INTO #temp(data) values (N'ハロウィンの飾りつけになっているJVCケンウッド丸の内ショールームに行ってきました。')

SELECT * FROM #temp

ASP.NET - Pass objects between server and client- using http handler, Ajax and JSON

This article shows an example of how to pass objects from client to server and from server to client. This helps to maintain state of object between server and client. Make sure you do not store sensitive data on client side.

1) Create a web application

2) Define a class in C#

   public class MyData

    {

        public int Prop1 { get; set; }

        public int Prop2 { get; set; }

        public int Result { get; set; }

    }

3) Create a web form and define Javascript class which matches C# class

MyData = function (prop1, prop2) {

            this.Prop1 = prop1,

            this.Prop2 = prop2

            this.Result = 0;

        }

4) Create request handler to handle Ajax requests

  public void ProcessRequest(HttpContext context)

        {

            string action = context.Request.QueryString["action"];

            string input;

            using (StreamReader sr = new StreamReader(context.Request.InputStream))

            {

                input = sr.ReadToEnd();

            }

            JavaScriptSerializer serializer = new JavaScriptSerializer();

            //Get object from client

            MyData data =(MyData)serializer.Deserialize(input,typeof(MyData));

            if (action == "add")

            {

                data.Result = data.Prop1 + data.Prop2;

            }

            context.Response.ContentType = "text/json";

            //Send object back to client

            context.Response.Write(serializer.Serialize(data));

        }

        public bool IsReusable

        {

            get { return true; }

        }

5) Call the server http handler from web form. Given below is the full javascript

<script src="Scripts/jquery-1.4.1.min.js" type="text/javascript"></script>

    <script>

        MyData = function (prop1, prop2) {

            this.Prop1 = prop1,

            this.Prop2 = prop2

            this.Result = 0;

        }

        var myobject = new MyData(100, 2);

        $.ajax({ url: "RequestHandler.aspx?action=add",

            data: JSON.stringify(myobject),//Send object to server

            type: "POST",

            success: function (data) {//Get object from server

                alert(data.Result);

            }

        });

    </script>

6) Register http handler using web.config

<system.web>

    <httpHandlers>

      <add verb="GET,POST" path="RequestHandler.aspx" type="HttpHandlerTest.RequestHandler"/>

    </httpHandlers>

    <compilation debug="true"/>

  </system.web>

If you are running IIS7 you may have to register the handler as given below

<system.webServer>

 <handlers>   <add name="ajaxHandler" verb="GET,POST" path="RequestHandler.aspx" type="HttpHandlerTest.RequestHandler"/>

  </handlers>

</system.webServer>

Run the application and you can see object values being set in client side and passed to server. After server sets “Result” in object it is passed back to client.  

ASP.NET – Http handlers and AJAX

Given below is an example of implementing AJAX in asp.net using http handlers. Custom http handler with AJAX allows developers to add lot of features to classic ASP.NET application without postbacks. I personally find this way of development better than the MVC model.

Let’s implement an example. I used VS 2010 and .net 4.0

------------------------------------------------

1) Create a new web application – HttpHandlerTest

2) Create a new class file –‘RequestHandler’ . Make sure that the class implements “IHttpHandler”

    public class RequestHandler : IHttpHandler

    {

        public void ProcessRequest(HttpContext context)

        {

            string action = context.Request.QueryString["action"];

            string input;

            using (StreamReader sr = new StreamReader(context.Request.InputStream))

            {

                input = sr.ReadToEnd();

            }

            int result = 0;

            if (action == "add")

            {

                result = Int32.Parse(input.Split(new char[] { ',' })[0]) + Int32.Parse(input.Split(new char[] { ',' })[1]);

            }

            context.Response.ContentType = "text/json";

            context.Response.Write(result.ToString());

        }

        public bool IsReusable

        {

            get { return true; }

        }

    }

3) Create an aspx page to call the handler. Enter below JS to the page. Make sure that you have reference to jQuery

  <script src="Scripts/jquery-1.4.1.min.js" type="text/javascript"></script>

    <script>

        $.ajax({ url: "RequestHandler.aspx?action=add",

            data: "10,11",

            type: "POST",

            success: function (data) {

                alert(data);

            }

        });

    </script>

4) Register handler in web.config

<system.web>

    <httpHandlers>

      <add verb="GET,POST" path="RequestHandler.aspx" type="HttpHandlerTest.RequestHandler"/>

    </httpHandlers>

    <compilation debug="true"/>

 </system.web>

If you are running IIS7 you may have to register the handler as given below

<system.webServer>

  <handlers>

    <add name="ajaxHandler" verb="GET,POST" path="RequestHandler.aspx" type="HttpHandlerTest.RequestHandler"/>

  </handlers>

</system.webServer>

That’s it. Now run the application and you can see the result from handler as a JS alert message.

 --------------------------------

Wasn't that easy. Let me know what you think.  Not just about this article, but also about the blog if you have read the other articles I posted.

Happy coding 

ASP.NET - Render images dynamically with text overlay

This example shows how to render an image dynamically using aspx page with text overlay on top of it.


1) Create an aspx webform

2) Create a method as given below

public Bitmap OverlayText(Image oImage, string text)

        {

            Bitmap oBitmap = new Bitmap(oImage.Width, oImage.Height, System.Drawing.Imaging.PixelFormat.Format24bppRgb);

            Graphics oGraphics = Graphics.FromImage(oBitmap);

            oGraphics.DrawImage(oImage, 0, 0);

            Font f = new Font(new FontFamily("Arial"), 20);

           

            SolidBrush oBrush = new SolidBrush(Color.FromArgb(255, System.Drawing.ColorTranslator.FromHtml("#000000")));

            RectangleF rect = new RectangleF(10, oImage.Height - 150, oImage.Width - 50, 100);

            StringFormat strFormat = StringFormat.GenericTypographic;

            oGraphics.DrawString(text, f, oBrush, rect, strFormat);

            oGraphics.Dispose();

            oBrush.Dispose();

            f.Dispose();

            return oBitmap;

        }

3) Call the method as given below in page_load by passing in the image object and overlay text. Make sure that you specify the virtual path of the image.

System.Drawing.Image img = System.Drawing.Image.FromFile(Server.MapPath("[SET PATH OF IMAGE HERE]"));

            Bitmap outputimage =OverlayText(img, overlaytext);

            using (MemoryStream ms = new MemoryStream())

            {

                outputimage.Save(ms, System.Drawing.Imaging.ImageFormat.Png);

                ms.WriteTo(Response.OutputStream);

            }

            Response.ContentType = "image/png";

  

That's it. Run the app and you will see the text overlay on the image

Parallel Task in C#

I was researching parallel task in C# and found that parallelism can be achieved in different ways, mainly by using "Task" and "Parallel" methods. But internally "parallel" uses "task" to do the process.

Anyway given below are some ways to run parallel tasks in C#

Create a class called 'Process'

public class Process1

{

    public int Add(int i, int j,int k)

    {

        return i + j + k;

    }

}

Given below are different ways to call the "Add" method in parallel

/*Parallel processing using "Task"*/

Process1 process1 = new Process1();

Task<int> task1 = Task<int>.Factory.StartNew(() => process1.Add(10, 11, 4));

Task<int> task2 = Task<int>.Factory.StartNew(() => process1.Add(29, 1, 2));

Response.Write(task1.Result.ToString() + "  " + task2.Result.ToString());

/*Parallel processing using "Parallel.ForEach"*/

List<Tuple<int, int, int>> data = new List<Tuple<int, int, int>>();//Generic list for Tuple

data.Add(new Tuple<int, int, int>(10, 11, 4));//add data to list

data.Add(new Tuple<int, int, int>(29, 1, 2));

ParallelLoopResult result = Parallel.ForEach(data, item => {//Para

int result1 = process1.Add(item.Item1, item.Item2, item.Item3);

Response.Write("Parallel result "+result1.ToString());

});

/*Parallel processing using "Parallel.Invoke"*/

Parallel.Invoke(

() => { int result1 = process1.Add(10, 11, 4); Console.Write(result1.ToString()); },

() => { int result1 = process1.Add(29, 1, 2); Console.Write(result1.ToString()); }

);

Run installations/apps remotely using Batch File

Today I had to install an app on multiple machines. So I used the below batch statement to copy and install the application to all machines without logging in to each box.

To use the below batch file you need psexec executable which can be downloaded from http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx

So how do you install apps on multiple machines without logging into each box?

1) Create test_copy.bat file with below statements. Replace [MSIFILE] with the installation file name

echo off
set arg1= %arg1%
xcopy /r /y "[MSIFILE].msi" \\"%arg1%"\e$\
psexec \\%arg1% cmd.exe /c msiexec /i e:\[MSIFILE].msi /L*v install.log /qn
del \\"%arg1%"\e$\[MSIFILE].msi
echo

2) Run "test_copy.bat" file from cmd line by specifying the machine name like "test_copy.bat -[machinename]"

That's it. You may have to provide additional parameters to "msiexec" statements to provide default installation directory etc. Also the windows user should have admin privilege on the installation box.

You can even run remote applications this way.