Windows Mobile: watch the memory footstep of running processes

Some times ago I posted my remote cpu usage monitor. Now here is a similar tool but for logging the memory. You can now watch the memory usage of processes remotely for example when you test an application.

There are two tools: vmUsage and vmUsageRecvr. You may use the mobile vmUsage alone and just use its logging. The other tool receives the memory status information on a PC and enables long time logging and export to a csv text.

vmusage   memeater-vm   excel-linechart

vmUsage is the mobile application that shows you a list of bars, one bar for each of the 32 possible process slots. It also shows the process name running in a slot and the memory usage. The memory usage is queried using a KernelIOCtl (IOCTL_KLIB_GETPROCMEMINFO). I found that API call at CodeProject. I first tried with the approach made in VirtualMemory at codeproject. But using VirtualQuery for every 4K block inside 32 pieces of 32MB takes a lot of time (256000 blocks!). The following shows a process memEater that is gaining more and more memory:

memeater-vmusage   memeater-vmusage2

You can also see the total physical and available memory in the first bar and you will recognize irregular memory changes too.

The small tool sends all data using UDP to possible receivers. My receiver is called vmUsageRecvr and receives the data and saves every virtual memory status set it to a SQLite database. The data can then be exported and is re-arranged by known processes. The live view of vmUsageRecvr shows the latest receive memory status and a small line graphic showing how the total used memory changed over time.

You can use the exported data in excel and again produce nice graphics.

excel-vmusage

In the above graph you can see memeater is consuming memory in 1MB steps until it crashes. The other memory peek is produced by pimg.exe, the camera dialog, when I made a photo.

Processes may start and go and so there process ID will be zero when they are not running. If a process is gone, vmUsage will not record it in its log:

20130221 06:17 pimg    569344    VMusage.exe    2105344    MemEater.exe    15175680    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    323395584    vtotal    33554432tvfree    26476544    load    35    
20130221 06:17 pimg    569344    VMusage.exe    2105344    MemEater.exe    16228352    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    322342912    vtotal    33554432tvfree    26476544    load    35    
20130221 06:17 pimg    569344    VMusage.exe    2105344    MemEater.exe    17281024    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    322342912    vtotal    33554432tvfree    26476544    load    35    
20130221 06:17 pimg    569344    VMusage.exe    2105344    MemEater.exe    17281024    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    321282048    vtotal    33554432tvfree    26476544    load    35    
20130221 06:17 pimg    569344    VMusage.exe    2105344    MemEater.exe    18337792    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    320163840    vtotal    33554432tvfree    26476544    load    36    
20130221 06:17 pimg    569344    VMusage.exe    2105344    MemEater.exe    19456000    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    320163840    vtotal    33554432tvfree    26476544    load    36    
20130221 06:17 pimg    569344    VMusage.exe    2105344    MemEater.exe    19456000    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    319111168    vtotal    33554432tvfree    26476544    load    36    
20130221 06:17 pimg    569344    VMusage.exe    2109440    MemEater.exe    20508672    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    318054400    vtotal    33554432tvfree    26476544    load    36    
20130221 06:17 pimg    569344    VMusage.exe    2109440    MemEater.exe    21561344    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    317001728    vtotal    33554432tvfree    26476544    load    36    
20130221 06:17 pimg    569344    VMusage.exe    2109440    MemEater.exe    22614016    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    317001728    vtotal    33554432tvfree    26476544    load    36    
20130221 06:17 pimg    569344    VMusage.exe    2109440    MemEater.exe    22614016    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    315949056    vtotal    33554432tvfree    26476544    load    37    
20130221 06:17 pimg    569344    VMusage.exe    2109440    MemEater.exe    23666688    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    314896384    vtotal    33554432tvfree    26476544    load    37    
20130221 06:17 pimg    569344    VMusage.exe    2109440    MemEater.exe    24719360    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    314896384    vtotal    33554432tvfree    26476544    load    37    
20130221 06:17 pimg    569344    VMusage.exe    2109440    MemEater.exe    24719360    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    313843712    vtotal    33554432tvfree    26476544    load    37    
20130221 06:17 pimg    569344    VMusage.exe    2109440    MemEater.exe    25772032    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    314822656    vtotal    33554432tvfree    27459584    load    37    
20130221 06:18 pimg    569344    VMusage.exe    1191936    MemEater.exe    25772032    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    340561920    vtotal    33554432tvfree    27328512    load    32    
20130221 06:18 pimg    569344    VMusage.exe    1323008    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    340430848    vtotal    33554432tvfree    27197440    load    32    
20130221 06:18 pimg    569344    VMusage.exe    1388544    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    340365312    vtotal    33554432tvfree    27131904    load    32    
20130221 06:18 pimg    569344    VMusage.exe    1519616    tmail.exe    303104    IQueue.exe    679936    total    493723648    free    340234240    vtotal    33554432tvfree    27000832    load    32

 

When a process is gone in vmUsageRecr, the process data is still there. In the following export viewed in excel you can see pimg is first not running. After pimg is started it consumes about 368KB. Then I took a photo and the memory increased to 1.1MB:

excel_process_0

Notes about the code

As said, the code is similar to my cpuMon tool.

A costum panel to simulate a bar graph

vmUsage uses a large panel and places 32 smaller custom panels inside to display a bar graphic. The custom panel’s background is just green and I draw a rectangle on top to visualize the current value. Alternatively I could have used 32  progress bars but I liked to also have text inside the graphic. Here is the OnPaint override code:

        protected override void OnPaint(PaintEventArgs e)
        {
            //draw the background rectangle
            e.Graphics.FillRectangle(new SolidBrush(BackColor), 0, 0, (int)((float)(this.Width / _Maximum) * _Maximum), this.Height);
            //draw foreground rectangle
            if (scaleMode == scaleModeValue.relative)
            {
                e.Graphics.FillRectangle(new SolidBrush(ForeColor), 0, 0, (int)((float)(this.Width / _Maximum) * @Value), this.Height);
            }
            else
            {
                e.Graphics.FillRectangle(new SolidBrush(ForeColor), 0, 0, (int)@Value, this.Height);
            }
            //draw text
            if (Text.Length > 0)
            {
                StringFormat sf = new StringFormat();
                sf.Alignment=StringAlignment.Center;
                RectangleF rect = new RectangleF(0f, 0f, this.Width, this.Height);
                e.Graphics.DrawString(Text, base.Font, new SolidBrush(Color.Black), rect, sf);
            }
            base.OnPaint(e);
        }

When vmUsage is started it starts a background thread that captures the current memory usage data:

            //start the background tasks
            vmiThread = new vmInfoThread();
            vmiThread._iTimeOut = iTimeout*1000;
            vmiThread.updateEvent += new vmInfoThread.updateEventHandler(vmiThread_updateEvent);

In the background thread I am using two events and a queue to sync foreground and background working:

        public vmInfoThread()
        {
            _fileLogger = new Logging.fileLogger(Logging.utils.appPath + "vmusage.log.txt");

            eventEnableCapture = new AutoResetEvent(true);
            eventEnableSend = new AutoResetEvent(false);

            //procStatsQueue = new Queue<ProcessStatistics.process_statistics>();
            procStatsQueueBytes = new Queue<byte[]>();

            myThreadSocket = new Thread(socketThread);
            myThreadSocket.Start();

            myThread = new Thread(usageThread);
            myThread.Start();
        }

The queue is used to decouple the data capture and the data send functions. One thread captures the data into a queue and then releases the socket thread which reads the queued data and releases the data capture thread:

        /// <summary>
        /// build thread and process list periodically and fire update event and enqueue results for the socket thread
        /// </summary>
        void usageThread()
        {
            try
            {
                int interval = 3000;
                //rebuild a new mem usage info
                VMusage.CeGetProcVMusage vmInfo = new CeGetProcVMusage();

                while (!bStopMainThread)
                {
                    eventEnableCapture.WaitOne();
                    List<VMusage.procVMinfo> myList = vmInfo._procVMinfo; //get a list of processes and the VM usage
                    StringBuilder sbLogInfo = new StringBuilder();  //needed to merge infos for log

                    System.Threading.Thread.Sleep(interval);
                    uint _totalMemUse = 0;
                    long lTimeStamp = DateTime.Now.ToFileTimeUtc();

                    //send all data in one block
                    List<byte> buffer = new List<byte>();
                    buffer.AddRange(ByteHelper.LargePacketBytes);
                    foreach (VMusage.procVMinfo pvmi in myList)
                    {
                        pvmi.Time = lTimeStamp;
                        buffer.AddRange(pvmi.toByte());

                        _totalMemUse += pvmi.memusage;

                        if (!pvmi.name.StartsWith("Slot", StringComparison.InvariantCultureIgnoreCase))
                        {
                            //_fileLogger.addLog(pvmi.ToString());    //adds one row for each VM info
                            sbLogInfo.Append(pvmi.name + "\t" + pvmi.memusage.ToString() + "\t");
                        }
                    }                    
                    procStatsQueueBytes.Enqueue(buffer.ToArray());

                    onUpdateHandler(new procVMinfoEventArgs(myList, _totalMemUse));

                    //send MemoryStatusInfo
                    memorystatus.MemoryInfo.MEMORYSTATUS mstat = new memorystatus.MemoryInfo.MEMORYSTATUS();
                    if (memorystatus.MemoryInfo.GetMemoryStatus(ref mstat))
                    {
                        MemoryInfoHelper memoryInfoStat= new MemoryInfoHelper(mstat);

                        //send header
                        procStatsQueueBytes.Enqueue(ByteHelper.meminfostatusBytes);
                        //send data
                        procStatsQueueBytes.Enqueue(memoryInfoStat.toByte());

                        //log global memstatus
                        sbLogInfo.Append(
                            "total\t" + memoryInfoStat.totalPhysical.ToString() +
                            "\tfree\t" + memoryInfoStat.availPhysical.ToString() +
                            "\tvtotal\t" + memoryInfoStat.totalVirtual.ToString() +
                            "tvfree\t" + memoryInfoStat.availVirtual.ToString() +
                            "\tload\t" + memoryInfoStat.memoryLoad + "\t");
                    }

                    //write a log line
                    _fileLogger.addLog(sbLogInfo.ToString()+"\r\n");
                    procStatsQueueBytes.Enqueue(ByteHelper.endOfTransferBytes);
                    ((AutoResetEvent)eventEnableSend).Set();
                }//while true
            }
            catch (ThreadAbortException ex)
            {
                System.Diagnostics.Debug.WriteLine("ThreadAbortException: usageThread(): " + ex.Message);
            }
            catch (Exception ex)
            {
                System.Diagnostics.Debug.WriteLine("Exception: usageThread(): " + ex.Message);
            }
            System.Diagnostics.Debug.WriteLine("Thread ENDED");
        }

The call eventEnableCapture.WaitOne(); waits until the event is set by the socket thread. Immediately after this call a new memory usage dataset is loaded. The rest of the code converts the data to bytes and adds the byte buffer to a queue. Then another event is set to release the socketThread.

        /// <summary>
        /// send enqueued objects via UDP broadcast
        /// </summary>
        void socketThread()
        {
            System.Diagnostics.Debug.WriteLine("Entering socketThread ...");
            try
            {
                const int ProtocolPort = 3002;
                sendSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
                sendSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.Broadcast, 1);
                sendSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.SendBuffer, 32768);

                IPAddress sendTo = IPAddress.Broadcast;// IPAddress.Parse("192.168.128.255");  //local broadcast
                EndPoint sendEndPoint = new IPEndPoint(sendTo, ProtocolPort);

                //UdpClient udpC = new UdpClient("255.255.255.255", 1111);
                System.Diagnostics.Debug.WriteLine("Socket ready to send");

                while (!bStopSocketThread)
                {
                    //block until released by capture
                    eventEnableSend.WaitOne();
                    lock (lockQueue)
                    {
                        //if (procStatsQueue.Count > 0)
                        while (procStatsQueueBytes.Count > 0)
                        {
                            byte[] buf = procStatsQueueBytes.Dequeue();
                            if (ByteHelper.isEndOfTransfer(buf))
                                System.Diagnostics.Debug.WriteLine("sending <EOT>");

                            sendSocket.SendTo(buf, buf.Length, SocketFlags.None, sendEndPoint);
                            System.Diagnostics.Debug.WriteLine("Socket send " + buf.Length.ToString() + " bytes");
                            System.Threading.Thread.Sleep(2);
                        }
                    }
                    ((AutoResetEvent)eventEnableCapture).Set();
                }

            }
            catch (ThreadAbortException ex)
            {
                System.Diagnostics.Debug.WriteLine("ThreadAbortException: socketThread(): " + ex.Message);
            }
            catch (Exception ex)
            {
                System.Diagnostics.Debug.WriteLine("Exception: socketThread(): " + ex.Message);
            }
            System.Diagnostics.Debug.WriteLine("socketThread ENDED");
        }

The first call after eventEnableSend.WaitOne(); inside the socket thread locks the queue, so no other code is able to access it. This is not necessary here as we sync the access using the events but I left the lock to show an alternative for async queue access. The rest of the code inside socket thread just sends the bytes of the queue using UDP.

Minor protocol between sender and receiver

During the development of the code I found it better to send a large block of bytes instead of small packets sending each memory info separately. After some time I added some special packets to mark the end-of-transfer. These are usefull to let the receiver know about which packets will come. So I added a memoryStatusInfo packet that informs the receiver when a memory status packet is attached in contrast to the virtual memory information block.

    void MessageReceivedCallback(IAsyncResult result)
    {
        EndPoint remoteEndPoint = new IPEndPoint(0, 0);
        try
        {
            //all data should fit in one package!
            int bytesRead = receiveSocket.EndReceiveFrom(result, ref remoteEndPoint);
            //System.Diagnostics.Debug.WriteLine("Remote IP: " + ((IPEndPoint)(remoteEndPoint)).Address.ToString());

            byte[] bData = new byte[bytesRead];
            Array.Copy(recBuffer, bData, bytesRead);
            if (ByteHelper.isEndOfTransfer(bData))
            {
                System.Diagnostics.Debug.WriteLine("isEndOfTransfer");
                updateEndOfTransfer();// end of transfer
            }
            else if (ByteHelper.isMemInfoPacket(bData))
            {
                System.Diagnostics.Debug.WriteLine("isMemInfoPacket");
                try
                {
                    VMusage.MemoryInfoHelper mstat = new VMusage.MemoryInfoHelper();
                    mstat.fromByte(bData);

                    //System.Diagnostics.Debug.WriteLine(mstat.ToString());

                    updateMem(mstat);
                }
                catch (Exception) { }
            }
            else if(ByteHelper.isLargePacket(bData)){
                System.Diagnostics.Debug.WriteLine("isLargePacket");
                try
                {
                    List<procVMinfo> lStats = new List<procVMinfo>();
                    VMusage.procVMinfo stats = new VMusage.procVMinfo();
                    lStats = stats.getprocVmList(bData, ((IPEndPoint)(remoteEndPoint)).Address.ToString());
                    updateStatusBulk(lStats);
                    //foreach (procVMinfo pvmi in lStats)
                    //{
                    //    pvmi.remoteIP = ((IPEndPoint)(remoteEndPoint)).Address.ToString();
                    //    //System.Diagnostics.Debug.WriteLine( stats.dumpStatistics() );
                    //}
                }
                catch (Exception) { }

            }
            else
            {
                System.Diagnostics.Debug.WriteLine("trying vmUsagePacket...");
                try
                {
                    VMusage.procVMinfo stats = new VMusage.procVMinfo(bData);
                    stats.remoteIP = ((IPEndPoint)(remoteEndPoint)).Address.ToString();
                    //System.Diagnostics.Debug.WriteLine( stats.dumpStatistics() );
                    if (stats.Time == 0)
                        stats.Time = DateTime.Now.ToFileTimeUtc();
                    updateStatus(stats);
                }
                catch (Exception) { }
            }
        }
        catch (SocketException e)
        ...

In the above vmUsage Recvr code you can see the different branches for different packet types: EndOfTransfer, MemInfoPacket and isLargePacket.

Network only knows bytes

Using TCP/IP you can only transfer bytes and so my memory info classes all contain code to convert from to bytes – a basic serialization and de-serialization. The class files are shared between the Windows Mobile vmUsage and the Windows vmUsageRecvr code. Following is an example of the VMInfo class.

    /// <summary>
    /// holds the VM data of one process
    /// </summary>
    public class procVMinfo
    {
        public string remoteIP;
        public string name;
        public UInt32 memusage;
        public byte slot;
        public UInt32 procID;
        public long Time;
...
        public byte[] toByte()
        {
            List<byte> buf = new List<byte>();
            //slot
            buf.AddRange(BitConverter.GetBytes((Int16)slot));
            //memusage
            buf.AddRange(BitConverter.GetBytes((UInt32)memusage));
            //name length
            Int16 len = (Int16)name.Length;
            buf.AddRange(BitConverter.GetBytes(len));
            //name string
            buf.AddRange(Encoding.UTF8.GetBytes(name));
            //procID
            buf.AddRange(BitConverter.GetBytes((UInt32)procID));
            //timestamp
            buf.AddRange(BitConverter.GetBytes((UInt64)Time));

            return buf.ToArray();
        }
        public procVMinfo fromBytes(byte[] buf)
        {
            int offset = 0;

            //is magic packet?
            if (ByteHelper.isLargePacket(buf))
                offset += sizeof(UInt64);   //cut first bytes

            //read slot
            this.slot = (byte)BitConverter.ToInt16(buf, offset);
            offset += sizeof(System.Int16);

            UInt32 _memuse = BitConverter.ToUInt32(buf, offset);
            memusage = _memuse;
            offset += sizeof(System.UInt32);

            Int16 bLen = BitConverter.ToInt16(buf, offset);
            offset += sizeof(System.Int16);
            if (bLen > 0)
            {
                this.name = System.Text.Encoding.UTF8.GetString(buf, offset, bLen);
            }
            offset += bLen;
            this.procID = BitConverter.ToUInt32(buf, offset);

            offset += sizeof(System.UInt32);
            this.Time = (long) BitConverter.ToUInt64(buf, offset);

            return this;
        }

You see a lot of BitConverter calls. The fromByte function needs to keep track of the offset for reading following data.

vmUsageRecvr

The code uses also a queue to transfer data between background thread (RecvBroadcast) and the GUI.

        public frmMain()
        {
            InitializeComponent();
            //the plot graph
            c2DPushGraph1.AutoAdjustPeek = true;
            c2DPushGraph1.MaxLabel = "32";
            c2DPushGraph1.MaxPeekMagnitude = 32;
            c2DPushGraph1.MinPeekMagnitude = 0;
            c2DPushGraph1.MinLabel = "0";

            dataQueue = new Queue<VMusage.procVMinfo>();

            dataAccess = new DataAccess(this.dataGridView1, ref dataQueue);

            recvr = new RecvBroadcst();
            recvr.onUpdate += new RecvBroadcst.delegateUpdate(recvr_onUpdate);
            recvr.onUpdateBulk += new RecvBroadcst.delegateUpdateBulk(recvr_onUpdateBulk);
            recvr.onEndOfTransfer += new RecvBroadcst.delegateEndOfTransfer(recvr_onEndOfTransfer);

            recvr.onUpdateMem += new RecvBroadcst.delegateUpdateMem(recvr_onUpdateMem);
        }

As we have different packet types for global memory status, single and bulk virtual memory data, I implemented multiple delegates. One handler of is the bulk updater. It gets its data via the custom event arg which is a list of all virtual memory dat for all ‘slots’:

        void recvr_onUpdateBulk(object sender, List<VMusage.procVMinfo> data)
        {
            foreach (VMusage.procVMinfo pvmi in data)
                addData(pvmi);
        }

The data is then feed into the GUI using the addData call:

        delegate void addDataCallback(VMusage.procVMinfo vmdata);
        void addData(VMusage.procVMinfo vmdata)
        {
            if (this.dataGridView1.InvokeRequired)
            {
                addDataCallback d = new addDataCallback(addData);
                this.Invoke(d, new object[] { vmdata });
            }
            else
            {
                dataGridView1.SuspendLayout();
                //enqueue data to be saved to sqlite
                dataQueue.Enqueue(vmdata);

                if (bAllowGUIupdate)
                {
                    dataAccess.addData(vmdata);
                    //release queue data
                    dataAccess.waitHandle.Set();
                }
                dataGridView1.Refresh();
                dataGridView1.ResumeLayout();
            }
        }

You see we have to prepare for cross event calling. Then we suspend the refreshing of the datagrid. The sql data is updated using a queue to decouple the GUI and SQL data saving. dataAccess.addData adds the data to the dataset that is bound to the datagrid.

There is lot more of code inside, just take a look if you like.

Questions?

Leave me a comment if you have any questions.

Source Code

Source code can be loaded from github

Leave a Reply