Main idea Key point it to understand the below pictures Iteration steps for each input, generator G outputs for each output, calculate logits_prob for each token in current, old, referenc...

LLM Application The purpose of Appcalition is to try our best to meed the demand of users. Depth of technology is not very important. There are many unexplainable points in the field of LLM, as a...

LLM Inference It’s a great milestone when I finish a mature and iterable project, even it’s still in the first stage. There’re plenty AI infra repositoreis and projects in github, but most of the...

This is the first week to work as a LLM inference engineer, the real work content is what I want in the past. I don’t have much experience in this field, But it’s still acceptable. To do what I l...

This is the second time period that I need to seek a job position in my life till now. I don’t have much experience to take a interview in the past, I think I learn a lot this time. First lesson I...

Happy New Year! Feel more happiness this year, really enjoy.

Product AutoSwitch Translate Final make this extension, which can automatically switch target language when source language is same with target language. What surprised me was I only spen half day...

What I do keywods: Jan-May: work Travel: wuhan, huangshan IELTS Products: Chinese blog collection slack to discord IELTS speaking assistant PopTranslate ...

Product Triton Try to debug the Liger-Kernel but found it works well except bfloat type. Learning news about LLM translation and try to realise the whole one in this kind of project, like cut cros...

Product Full Cross Entropy Realising of fast cross entropy Based on previous knowledge on fast cross-entropy, realizing it by triton doesn’t spend too much. There are only 1e-7 difference between...