[ZenQ]

Hi,
I am using AForge.Net framework to do some video processing.
What I am doing is:
1 - Browse Video File(.AVI format) or Connect to Video Capture Device
2 - Display the original video in picturebox1
3 - manipulate the video into black and white and display in picturebox2
4 - count zero and one from manipulated video and display sum in textbox1

So far I have done the 1st until 3rd steps

I am now stuck on the 4th step
What I understand:
i - get img from video and store them in a variable
ii - loop to check every pixel color from stored img
iii- display into textbox

Here is the code, it worked when I tested them on a picture. but failed when testing on a video.

private void binaryImg(Bitmap bimg, int x , int y)
        {
         for (int i = 0; i < bimg.Height; i++)
            {
                for (int j = 0; j < bimg.Width; j++)
                {
                    if (bimg.GetPixel(j,i).B.ToString() == "255" && bimg.GetPixel(j,i).G.ToString() == "255" && bimg.GetPixel(j,i).R.ToString() == "255")
                        x += 1;

                    else
                        y += 1;

                }
                textBox1.Text = "ONE: " + x + "\n\rTWO: " + y;
            }
}

The bimg was the black/white video stream..I used the edgedetector from AForge.Net

private void displayFrame(object sender, NewFrameEventArgs eventArgs)
        {
            picturebox1.Image = (Bitmap)eventArgs.Frame.Clone(); // original video
            Bitmap image = (Bitmap)eventArgs.Frame.Clone();
            Bitmap greyImg = Grayscale.CommonAlgorithms.BT709.Apply(image);
            SobelEdgeDetector filter = new SobelEdgeDetector();
            Bitmap edge = filter.Apply(greyImg);
            picturebox2.Image = (Bitmap)egde; //black/white video
            binaryImg(edge,one,zero);
            textBox1.Text = "ONE: " + one + "\n\rTWO : " + zero;//}
        }

The binaryImage took some time when I tested them to process a big picture, so how can I fix this?
What did I do wrong?

I also got an InvalidOperationException "Cross-thread operation not valid: Control 'textBox1' accessed from a thread other than the thread it was created on."
when I tried to use them on a video file/webcam.
Thank you.

[ragingben]

Quote

Here is the code, it worked when I tested them on a picture. but failed when testing on a video.

Are you doing this on a live video? Try capturing a frame of the video and doing all your processing on that.

Quote

I also got an InvalidOperationException "Cross-thread operation not valid: Control 'textBox1' accessed from a thread other than the thread it was created on."
when I tried to use them on a video file/webcam.

Check which thread is trying to acess the picturebox. If it is accessed incorrectly from a thread other than the UI thread it will throw this error. You need to use the Invoke, or BeginInvoke methods of the text box. I would create a delgate to use for updating the text...

public delegate void StringUpdaterDelegate(String s);

...and then call this passing it the string...

this.textBox1.Invoke(new StringUpdaterDelegate((String s) =>
{
    // set text
    this.textBox1.Text = s;
}), "WHATERVER STRING");

[ZenQ]

Quote

Are you doing this on a live video? Try capturing a frame of the video and doing all your processing on that.

I am doing this for both video file and live video(depends on user selection)
I did capture the frame using

Bitmap image = (Bitmap)eventArgs.Frame.Clone();

Quote

Check which thread is trying to acess the picturebox. If it is accessed incorrectly from a thread other than the UI thread it will throw this error. You need to use the Invoke, or BeginInvoke methods of the text box. I would create a delgate to use for updating the text...

public delegate void StringUpdaterDelegate(String s);

..and then call this passing it the string...

this.textBox1.Invoke(new StringUpdaterDelegate((String s) =>
{
    // set text
    this.textBox1.Text = s;
}), "WHATERVER STRING");

Thank you. It works when I use a live video, but the video is moving too slow, and the sum of one and zero are not counted..they stayed the same ONE:0; Zero:0;

[ragingben]

Sorry I see the line

Bitmap image = (Bitmap)eventArgs.Frame.Clone();

now

Did you realize you are setting the textBox1.Text in 2 places - the binaryImg method, and also in displayFrame?

As the displayFrame sets the TextBox.Text just after the the TextBox.Text is set in binaryImg meth, the change in binaryImg will probally be so fast you won't notice it.

You are then setting the TextBox.Text for the 1's and 0's with the variables one and zero, which you pass to the binaryImg method (where they are set) by value, so although the values are changed in the binaryImg metgod, the change is only valid within that method. If you intend it to work this way, you need to pass by reference, using the ref keyword. Or you could just comment out the line in the displayFrame method:

textBox1.Text = "ONE: " + one + "\n\rTWO : " + zero;//

But is that how you want this to work?

[ZenQ]

Ops my bad didn't saw that. Okay I fixed it and now it is working. However the problem now is the video is still moving too slow. Are there any way to improve the processing without affecting much of the videos fps? The video delay was around 4~5 seconds..

[ragingben]

I worked on a program at work that deals with video ovelay, the video coming from a black and white camera looking down a microscope, and I had to access the colour of the overlay based on the overall darkness of the video feed, without affecting the FPS. The way I did this is by taking a frame, getting a overall feel of the pixels in the frame (I won't bother going into this), and then outputting a value for the overall pixel colour in grayscale (0-255), and then adjusting the overlay so it is visible on any video frame (negative colour). I did this is WPF, but that is a bit irrelavant.

The point is this: I way only analyzing one frame at a time, and discarded all other frames from being anazlyed until that one frame was done with. Then I would analyze another, and so forth, always on a seperate thread from UI.

If you are doing that kind of process on each and every video frame you will get slow down. It is so much work for the processor.

Imagine a 800x600 video running at 25fps. In 8 bit grayscale that is 480kb per frame, over 25 frames that is around 12MB. Analzing 12MB of data every second, aswell as processing the video feed is generally going to cause a bit of slow down.

This post has been edited by ragingben: 14 March 2011 - 05:51 AM

[ragingben]

To elaborate on my point, do you need to process every frame?

[ZenQ]

What I am trying to do is like the one i posted some time here few weeks ago..while waiting for response I tried googling and seek help from my supervisor and friends, which lead me to where I am right now.

What I want to do originally is to grid the video into several segments(let say 4x4), then check differences between pixels from previous image, if there are changes in the binary(0 and 1) then that square grid give a value of 1, else it stays 0. I will then use these values as input neurons for either Back Propagation or Kohonen Self Organizing Map and do some calculation from there, the output will probably be something like:
"There is an object on the left" or something similar.

#NOTE# The video File / Live Video will be from a moving camera, so my scope of environment is for white surface/background to reduce noise from surroundings.

I do not know whether I will or should process every frame or not.
The problem I posted here was to see if I can proceed with using binaryImg or not, so it is a no no?

[ragingben]

Ah, OK I understand. So by splitting a video frame into segments, and comparing a segment to the same segment of the frame before you can detect if something has changed in the new frame, i.e an object moved?

If you can calculate each frame really depends on many factors; the pixel format of the framae, the width and height of the frame, the FPS, the host system...

I've noticed that by ensuring that R, G and B must all be 255, you are only incrementing X when the colour is pure white? Is this intended? Because this requires strong lighting to work correctly - maybe you could do with a threshold value - anything over say 240 class as white so 0, else a shade of gray (black) - 1?

Also, what happens if the pixels in the segment all change around (say you having speckeling if the sharpness is too high), you could get a result that has the same number of 1's and 0's even though the picture is totally different.

I would seriously recommend that your binaryImage method returns something as a result rather than changing text in a text box.

For example, this structure would be suitable:

/// <summary>
/// Represents the result of a binary image
/// </summary>
public struct BinaryImageResult
{
    #region Fields

    /// <summary>
    /// Get how many black pixels the binary image contains
    /// </summary>
    public Int32 BlackPixels
    {
        get { return this.blackPixels; }
        protected set { this.blackPixels = value; }
    }

    /// <summary>
    /// Get or set how many black pixels the binary image contains
    /// </summary>
    private Int32 blackPixels = 0;

    /// <summary>
    /// Get how many white pixels the binary image contains
    /// </summary>
    public Int32 WhitePixels
    {
        get { return this.whitePixels; }
        protected set { this.whitePixels = value; }
    }

    /// <summary>
    /// Get or set how many white pixels the binary image contains
    /// </summary>
    private Int32 whitePixels = 0;

    /// <summary>
    /// Get how many pixels the binary image contains
    /// </summary>
    public Int32 Pixels
    {
        get { return this.BlackPixels + this.WhitePixels; }
    }

    #endregion

    #region Methods

    /// <summary>
    /// Represents the result of a binary image
    /// </summary>
    /// <param name="blackPixels">The amount of </param>
    /// <param name="whitePixels"></param>
    public BinaryImageResult(Int32 blackPixels, Int32 whitePixels)
    {
        // set black
        this.BlackPixels = blackPixels;

        // ste white
        this.WhitePixels = whitePixels;
    }

    #endregion

    #region StaticMethods

    /// <summary>
    /// Analyze a bitmap image
    /// </summary>
    /// <param name="bmp">The bitmap to analyze</param>
    /// <param name="threshold">The threshold to use for white detection</param>
    /// <returns></returns>
    public static BinaryImageResult AnalyzeImage(Bitmap bmp, Int32 threshold)
    {
        // hold white pixels
        Int32 whites = 0;

        // itterate rows
        for (Int32 row = 0; row < bmp.Height; row++)
        {
            // itterate columns
            for (Int32 column = 0; column < bmp.Width; column++)
            {
                // get pixel
                Color pixel = bmp.GetPixel(column, row);

                // if white
                if (BinaryImageResult.isColourWhite(pixel, threshold))
                {
                    // increment whites
                    whites++;
                }
            }
        }

        // return result
        return new BinaryImageResult((bmp.Width * bmp.Height) - whites, whites);
    }

    /// <summary>
    /// Determine if a colour is white
    /// </summary>
    /// <param name="c">The colour to check</param>
    /// <param name="threshold">The threshold to use for white detection</param>
    /// <returns>true if the colour is white, false if it is gray or black</returns>
    protected static Boolean isColourWhite(Color c, Int32 threshold)
    { 
        return ((c.R >= (255 - threshold)) &&
                (c.G >= (255 - threshold)) &&
                (c.B >= (255 - threshold)));
    }

    #endregion
}

And then your binaryImage could be like...

private BinaryImageResult binaryImg(Bitmap bimg)
{
    return BinaryImageResult.AnalyzeImage(bimg, 25);
}

And the calling code can be as neat as this...

// get result
BinaryImageResult r = this.binaryImg(someImage);

// display info
this.textBox1.Text = String.Format("Black: {0}, White: {1}, Total: {2}", r.BlackPixels, r.WhitePixels, r.Pixels);

This post has been edited by ragingben: 14 March 2011 - 07:00 AM

[ragingben]

Sorry, the struct didn't compile, here is the bug-free version

/// <summary>
/// Represents the result of a binary image
/// </summary>
public struct BinaryImageResult
{
    #region Fields

    /// <summary>
    /// Get how many black pixels the binary image contains
    /// </summary>
    public Int32 BlackPixels
    {
        get { return this.blackPixels; }
        private set { this.blackPixels = value; }
    }

    /// <summary>
    /// Get or set how many black pixels the binary image contains
    /// </summary>
    private Int32 blackPixels;

    /// <summary>
    /// Get how many white pixels the binary image contains
    /// </summary>
    public Int32 WhitePixels
    {
        get { return this.whitePixels; }
        private set { this.whitePixels = value; }
    }

    /// <summary>
    /// Get or set how many white pixels the binary image contains
    /// </summary>
    private Int32 whitePixels;

    /// <summary>
    /// Get how many pixels the binary image contains
    /// </summary>
    public Int32 Pixels
    {
        get { return this.BlackPixels + this.WhitePixels; }
    }

    #endregion

    #region Methods

    /// <summary>
    /// Represents the result of a binary image
    /// </summary>
    /// <param name="blackPixels">The amount of </param>
    /// <param name="whitePixels"></param>
    public BinaryImageResult(Int32 blackPixels, Int32 whitePixels)
    {
        // set black
        this.blackPixels = blackPixels;

        // ste white
        this.whitePixels = whitePixels;
    }

    #endregion

    #region StaticMethods

    /// <summary>
    /// Analyze a bitmap image
    /// </summary>
    /// <param name="bmp">The bitmap to analyze</param>
    /// <param name="threshold">The threshold to use for white detection</param>
    /// <returns></returns>
    public static BinaryImageResult AnalyzeImage(Bitmap bmp, Int32 threshold)
    {
        // hold white pixels
        Int32 whites = 0;

        // itterate rows
        for (Int32 row = 0; row < bmp.Height; row++)
        {
            // itterate columns
            for (Int32 column = 0; column < bmp.Width; column++)
            {
                // get pixel
                Color pixel = bmp.GetPixel(column, row);

                // if white
                if (BinaryImageResult.isColourWhite(pixel, threshold))
                {
                    // increment whites
                    whites++;
                }
            }
        }

        // return result
        return new BinaryImageResult((bmp.Width * bmp.Height) - whites, whites);
    }

    /// <summary>
    /// Determine if a colour is white
    /// </summary>
    /// <param name="c">The colour to check</param>
    /// <param name="threshold">The threshold to use for white detection</param>
    /// <returns>true if the colour is white, false if it is gray or black</returns>
    private static Boolean isColourWhite(Color c, Int32 threshold)
    {
        return ((c.R >= (255 - threshold)) &&
                (c.G >= (255 - threshold)) &&
                (c.B >= (255 - threshold)));
    }

    #endregion
}

I bench tested the time in ms for this to get all pixel data, it is around 780ms for a 8.18Kb image, so it ain't too speedy My image is a 990x768 (sorry - weird size) png, so if I had 1 of these coming in a second the CPU could probally keep up, anymore and it is gunna choke up. Wether splitting into sements and handling on seperate threads will speed up the process is down to experimentation

This post has been edited by ragingben: 14 March 2011 - 07:21 AM

[ZenQ]

Quote

I've noticed that by ensuring that R, G and B must all be 255, you are only incrementing X when the colour is pure white? Is this intended? Because this requires strong lighting to work correctly - maybe you could do with a threshold value - anything over say 240 class as white so 0, else a shade of gray (black) - 1?

RGB is based on the video after it has already been processed into black/white(threshold?) as shown in the image.
So if white, increase value in X else in Y..

Quote

Also, what happens if the pixels in the segment all change around (say you having speckeling if the sharpness is too high), you could get a result that has the same number of 1's and 0's even though the picture is totally different.

I know, which is why the scope is white background/surface.
There was this one solution I found online(forgot) something about processing pixel through bytes instead of bits can increase performance? :s

Attached image(s)

• 

[ZenQ]

for (Int32 row = 0; row < bmp.Height; row++)
        {
            // itterate columns
            for (Int32 column = 0; column < bmp.Width; column++)
            {
                // get pixel
                Color pixel = bmp.GetPixel(column, row);

                // if white
                if (BinaryImageResult.isColourWhite(pixel, threshold))
                {
                    // increment whites
                    whites++;
                }
            }
        }

wont this just check the color of pixel from width to length?
Here is some code I saved for later part of the project. credit to my supervisor for some highlights

inX = 0;
inY = 0;
for (i = 0; i<9; i++) // let assume the grid is 3x3 as shown in image..so i iterate 9 times for each segments
   {
    val = 0;
    for(int y = inY; y<boundY; y++) //boundY refers to segment width boundary(the last pixel)
     {
      for(int x = inX; x<boundX; x++) //boundX refers to segment height boundary
        {
            img[j][val] = img[y][x]'
            val++;
        }
     }
   if(boundX>9)
     {
        boundY += 2;
        inY += 2;
        inX += 0;
        boundX = 2;
     }
    else
      {
          boundX += 2;
          inX += 2;
      }
}

*note*
the +2 in the if statement is just an example..the increment size is based on the number of pixel in each segments..
for a 9x9 grid, in 1 segment might have pixel of width 10(example) so it would be +=10; something like that.

I have not actually tried this since I am still stuck with changing the image into binary..but after your help, I think I can probably go to the next step(ignoring the processing performance)..

Attached image(s)

• 

This post has been edited by ZenQ: 14 March 2011 - 08:00 AM

[ragingben]

Hmmmm, that looks pretty full on to me, I have been playing around here and I have some segmentation comparing sorted for 2 bitmaps on the hard drive, it is really not that big a job as long as you plan correctly how you are going to do it. This is what I have:

1. Create a method in the BinaryImageResult struct that analyzes a Color[] and returns a BinaryImageResult
2. Create a method in the BinaryImageResult that looks at individual sections of the bmp (regions of interest), gets all the pixels and uses number 1 to return a BinaryImageResult[] representing the whole bitmap in segments
3. Make the BinaryImageResult struct implement the IComparable interface so that it can be compared to another BinaryImageResult.
4. By comparing each segment you can deduce where the change occured...

Does any of that make sense?

This post has been edited by ragingben: 14 March 2011 - 08:25 AM

[ragingben]

I have been playing with this a little more and I have created some classes and structs that do what you are asking, however the frame rate may be a little slow, but I will get to that in a minute...

Basically I figure that you are after three main things: A class that represents the bitmap in segments, A struct that represents a single segment, and an Interface for comparing these segments.Granted, you are going to need a little more than this in the long run, but this is enough to get you going.

(click the spoier tag if you want more info...)

Spoiler

A class that represents the bitmap in segments - SegmentedBitmap
This class should be "the daddy" - it should be able to take a bitmap, split it into however many segments that you are after, and then be able to determine which of these segments have changed when you compare it to another SegmentedBitmap.
I would recoomend the follwoing properties as a start:

• Columns
  
• Rows
  
• Segments

And these methods:

• DetermineChangedSegments()
  
• RebuildSource
  
• Update

And this static method:

• SegmentBitmap()

A struct that represents a single segment - BitmapSegment
This struct represents a segment of a bitmap image, and should be able to obtain an analysis of itself as a binary image, which is what you have stated you want to use. It should also know it's place alongside it's peers (other BitmapSegments), and it would be useful if it could construct itself into a bitmap if needed, but shouldn't be stored as one. By using struct instead of class (I think) it is held on the stack instead of the heap, saving lots of garbage collector work.
I would recoomend the follwoing properties as a start:

• Column
  
• Row
  
• Width
  
• Height
  
• Top
  
• Left
  
• Pixels

And these methods:

• AnalyseAsBinaryImage()
  
• GetAsBitmap

An Interface for comparing these segments - IBitmapAnalysis
This inteface is for analysing a bitmap image, and determining if it is simlar to another bitmap image
I would recoomend the follwoing fields as a start:

• BlackPixels
  
• WhitePixels
  
• Pixels

And thismethods:

• IsSimilarTo()

You could then create a variety of structs/classes that inherit from IBitmapAnalysis, which can provide different methods of comparing bitmaps. For exmple, a struct BinaryBitmapAnalysis could inherit from IBitmapAnalysis, and could contain an analysis of how many black and white pixels a provided has, and if this is silar to another BinaryBitmapAnalysis...

These three classes/structs are enough to compare 2 bitmap images for a change, provided your implementation is good. To run this with video you could use the SegmentedBitmap.Update() method to reuse the SegmentedBitmap rather than createing a new one, which is easier on the GC. You only ever need 2 SegmentedBitmap's to handle all frames coming in.

FPS could be a big issue for you, especially if you are aiming for 5 FPS +, but of course a lot depends on your CPU and the frame sizes. A frame of 10x10 would fly through this, but a 1024x768 frame is obviously going to choke up. You could look into lowering the resolution of the webcam to speed things up, or even lowering the resolution of the bitmap frames programatically if that fails, but that could end up taking up more processor than just dealing with the large frames in the first place. I'm not 100% sure, it is something you should look into though. Image analysis is never going to run as smoothly as just streaming the video.

I apologise if there are many spelling mistakes are bad grammer in that waffle

I hope you can find some sense and/or direction in there

This post has been edited by ragingben: 16 March 2011 - 03:46 AM

[ZenQ]

Thank you for the tips. I will begin working on them this weekends when i am free. Any update will be posted(hopefully I can do it flawlessly)..