tech

We recently caught a rare bug: One of our services is used to upload files to our storage service, and after uploading, it would check file format, and pass the extracted type to the next service. This file type inspection relies on python-magic. The bug occurs as it detected a

A recent project in our team was to improve the computing efficiency of a deep-learning-based video classifier. After carefully tuning the models and migrating to Kafka, we successfully reduced the cost by 84%. Background When we received the request, the model team already developed a POC version of the video

The first project after I joined this company was to help the team to deliver a document translator project. When I joined, the team already built a MVP version: it worked, but it was very slow. Translating a 50-page Microsoft Docx file would cost us more than 10 minutes, which

When I wrote my dissertation, I used Pandoc to convert my draft in markdown format to the final version in PDF format. Pandoc is an extremely powerful and easy-to-use tool for file conversion, as shown in the following diagram. When we load data into Python, we have a similar demand

CRF (Conditional Random Fields) has been a popular supervised learning method before deep learning occurred, and still, it is a easy-to-use and robust machine learning algorithm. We recently used this algorithm to do NER (name entity recognition), and here is a brief summary of using CRF in Python. Introduction of

With a focus on processing Chinese textual data, we have been using different tools extensively, namely Jieba and Stanford NLP software. Either of them has its own advantages and drawbacks. To balance efficiency and accuracy, we eventually chose Stanford NLP software as our default toolset. Therefore, this post introduces some