← Experience

Loop Harmony

Rebuilt a disaster and industrial site monitoring service in React, and added a video recording feature and MinIO storage and retrieval APIs to the existing Kurento server

P2ACH.AIApr 2024 – Aug 2024Freelance Developer

Role

Handled frontend and recording feature development on a 3-person outsourced team with the CEO and a backend developer

Problem

While rebuilding a disaster and industrial site monitoring platform in React, it had to support mobile and video recording, and WebRTC video showed up black only on iOS

Solution

  • DashboardIntegrated the map, WebRTC calls, chat and CCTV, and built a screen where widget layout can be adjusted with react-grid-layout
  • RecordingAdded a recording feature and MinIO storage and retrieval APIs to the existing Kurento server
  • iOS supportFixed the black screen with H.264 codec negotiation and autoplay settings

Result

Delivered PC and mobile monitoring screens, including iOS, and the recording feature

Overview

Loop Harmony is a platform that uses maps, CCTV video and real-time communication together for disaster monitoring and industrial sites. People can check the situation at a site from a PC, and users at the site can use calls and monitoring features on mobile and tablet.

Worked on the frontend and the recording feature in a 3-person outsourced team with the company representative and a backend developer. Developed the React migration of the existing service and the monitoring UI for each device, and added a video recording feature.

Key features

  • Integrated monitoring: Use voice and video calls and chat while viewing the map, CCTV video and screen sharing
  • Control screens for each device: Use monitoring features on mobile, tablet and PC
  • Dashboard layout: Move and resize widgets to arrange the information needed
  • Video recording: Record call video, then store and retrieve it
Loop Harmony PC monitoring dashboardPC control screen
Loop Harmony mobile Matrix video callMobile video call screen

Implementation and improvements

  1. React migration and monitoring UI for each device

    Moved the existing platform to React and laid out the screens for mobile, tablet and PC. Built a PC monitoring screen for viewing many pieces of information at once, and a mobile screen for checking calls and site information on a small display.

  2. Connecting calls, chat, map and video on one control screen

    Connected Rocket.Chat chat to calls based on WebRTC and Socket.IO, and linked a Leaflet map, CCTV streaming and screen sharing on the same monitoring screen. So that users can talk right away while looking at site location and video, screens that had been separate features were organized into one work flow.

  3. Fixing the iOS WebRTC black screen

    On iOS only, the other party's video showed as a black screen, a compatibility problem. Adjusted H.264 codec negotiation and autoplay settings so that call video displays on different devices too. Improved the monitoring screen by checking not only the layout but also actual video playback on each device.

  4. Adding recording to the Kurento server

    So that records can be kept on top of the call and video features, added a recording feature and MinIO storage and retrieval APIs to the existing Kurento server.

  5. A dashboard that users adjust

    The information and layout needed on screen can differ from user to user, so implemented widget moving and resizing with react-grid-layout. Users can set up the dashboard to suit their work.

Tech stack

Frontend

React · TypeScript · Tailwind CSS · shadcn/ui · React Query · Zustand · react-grid-layout

Real-time communication and server

WebRTC · Socket.IO · Node.js · Kurento · MinIO · Leaflet

Results and current status

Built a React UI that lets users use the monitoring features on one screen, and fixed the iOS WebRTC black screen problem with H.264 codec negotiation and autoplay settings. Delivered the PC and mobile monitoring screens, including iOS, and the recording feature.