lukkiddd // AI & ML

Paper: TrustLLM This research paper also contains the prompts in order to evaluate trustworthiness Guidelines and Principles for Trustworthiness Assessment of LLMs Truthfulness The accurate representation of information, facts, and r...

Paper: Mistral 7B Our model, Mistral 7B, demonstrates that a carefully designed language model can deliver high performance while maintaining an efficient inference. Mistral 7B outperforms the best open 13B model (Llama 2) across all evaluated benchmark...

Read handbook: Cognizant - Generative AI Handbook How are early pioneers applying GenerativeAI Expert assistants for professional work Generative AI assistants being used in legal software to augment services and accelerate work Me...

Hypothetical Document Embeddings (HyDE) Given a query, HyDE first zero-shot instructs an instruction-following language model (e.g. InstructGPT) to generate a hypothetical document. The document captures relevance patterns but is unreal and may contai...

RAG system aim to Reduce the problem of hallucinated responses from LLMs Link sources/references to generated responses Remove the need for annotating documents with meta-data. Takeaway Validation of a RAG system is only feasible during operatio...

Compositionality Gap - describe the fraction of compositional questions that the model answers incorrectly out of all the compositional questions for which the model answers the sub-questions correctly. Compositional questions require more computation and ...

Reference: https://arxiv.org/pdf/2401.04398.pdf Summary Working with table is hard Generic Reasoning, won't work, mostly due to difficulty in extractions Program-aided Reasoning, won't work, due to format of the data Propose Chain-of-thoug...

Reference: https://arxiv.org/pdf/2308.10335.pdf In short:Currently, software engineers should use tools like Co-pilot cautiously. The evaluation results show that even for GPT-4, 62% of the generated code contains API misuses, which would cause unexpected...

Top Tech News LandingAI - Introducing Domain-Specific Large Vision Models Example Hey Gen - AI Video Generator Papers Stable Diffusion XL Models CivitAI Juggernaut XL StableDiffusion Dall-E 3 MidJourney Tools/WebUI Stable Diff...

Resources for Large Language Model Prerequisite It would be easier if we have knowledge on NLP or Embedding, I would recommend Databricks course, the course start from the NLP knowledge till putting LLM to production. What is LLM LLM stands for Large Lan...

Vision Resources for Large Vision Model Language LLM Speech To explore

RAG Triad from Truera RAG Triad Answer Relevance Context Relevance Groundedness Answer Relevance from trulens_eval import Feedback f_qa_relevance = Feedback( provider.relevance_with_cot_reasons, name="Answer Relevance" ).on_i...

Summary OpenAI Function Calling Recent update from OpenAI to support Function calling import json # Example dummy function hard coded to return the same weather # In production, this could be your backend API or an external API def get_current_weather(l...

This is a cool course, which help you understand the tips, tactics to write a better prompt. Check it out ChatGPT Prompt Engineering for Developers Prompting Principles Principle 1: Write clear and specific instructions Tactic 1: Use delimiters to clearl...

Principal Solutions Architect for Generative AI at AWS ◆ Founder of NLP London Linkedin

RAG Essentials : How to split documents for embedding How to Create high-quality embeddings. How to choose vector store How to search in order to retrieve most useful and relevance document from embedding (vector store) How to retrieve the useful...

Whylogs help logs the data, as well as profiling and monitoring the data issue with LLM. In the note, we will cover Hallucination Data Leakage Refusals and Prompt Injections Passive and active monitoring 1. Hallucination Irrelevant Response o...

Summary: The LangChain Chat with Your Data course provides a step-by-step exploration of RAG Essentials, guiding users through fundamental processes. The course covers loading various document types using methods like website loaders, directory loaders, and...

After some learnings, I have tried to summarise my understanding into different sections, and write them in LLM Top Tech News Amazon Q Google Vertex Generative AI Studio Google Duet ChatGPT Enterprise Application Example kapa.ai mendable....

Summary: The LangChain for LLM Application Development Course, focuses on utilizing the LangChain Python/TypeScript framework to streamline the creation of Language Model (LLM) applications. The course highlights key processes specific to Retrieval-Augment...

Summary Author: William Byham My review: ⭐⭐⭐⭐ (4/5) It's a story about the castle (organization), that fight the dragon (problems) for their citizen (users). The story begin with Art Halegiver (main character) who are arrow-maker which bored with the...

[12] Supports COMMUNITY-BASED RECOMMENDER SYSTEMS: ANALYZING BUSINESS MODELS FROM A SYSTEM OPERATORS PERSPECTIVE Problem Baseline Result Reference Price and Profit Awareness in Recommender Systems Proble...

01 - Introduction to Causality When Association IS Causation Y0i is the potential outcome for unit i without treatment Y1i is the potential outcome for the same unit i with the treatment Y1i - Y0i is the potential outcomes of indiv...

How to pivot multiple columns in BigQuery SELECT * FROM ( SELECT date, event_name, user_id, count(*) as transactions FROM TABLE WHERE event_name IN ('impression', 'click') GROUP BY 1 ,2 ,3 ) PIVOT( COUNT(distinct user_...

Reference: https://www.techtarget.com/whatis/definition/VUCA-volatility-uncertainty-complexity-and-ambiguity What VUCA is an acronym that stands for volatility, uncertainty, complexity and ambiguity Use to create better understanding of the situations V...

Reference: Trust in Radical Truth and Radical Trasnparency Notes Realize that you have nothing to fear from knowing the truth Have integrity and demand it from others Create an environment in which everyone has the right to understand what makes sen...

The key is Candor. Reference: Braintrust at Pixar What is it Put smart, passionate people in a room together, charge them with identifying and solving problems, and encourage them to be candid. The Braintrust is not foolproof, but when we get it right...

Note Psychological safety: Can we take risks on this team without feeling insecure or embarrassed? Dependability: Can we count on each other to do high quality work on time? Structure & clarity: Are goals, roles, and execution plans on our t...

Manager's Actions Foster Psychological Safety Book - The Fearless Organization - Solutions section Resources The five keys to a successful Google team Book - The Fearless Organization

Summary Author: Amy Edmonson My review: ⭐⭐⭐⭐ (4/5) Ideas Driving goal with fear can hurt Silent can be dangerous Listen with empathy can help Solutions Setting the Stage Frame the work Emphasize Purpose Inviting Parti...

Authors of Book - The Fearless Organization

Authors of Book - Zapp - The lightning of empowerment Book - HeroZ Empower Yourself, Your Coworkers, Your Company All of his books

Summary Author: William Byham My review: ⭐⭐⭐⭐⭐ (5/5) Amazon: Book This is the first book I've read as a manager, which recommended by my first best manager ever!. The book tell a story about a company where employee has low engagement and burn out. On...

Writing Bullet Journal Zettelkasten Commonplace book

Measuring the long-term impact of product features is a challenge we often face. Traditional short-term experiments may hinder user experiences. However, a solution exists: holdback experiments. By assigning a small control group while keeping the majority ...

WITH engagement AS ( SELECT event_date ,user_id FROM engagement_table ), dau AS ( SELECT event_date ,COUNT(distinct user_id) AS count_dau FROM engagement GROUP BY 1 ) , string_aggregate AS ( SELECT event_date ,STRING_AGG(DISTINC...

Intervention to reduce churn Churn is hard to fight Churn is hard to prevent Churn is hard to predict Reducing churn is a team effort Great Customer Metrics Customers who use the product features more than five ...

Payments Type Recurring payments - Fixed payments of the same amount for each period of service Usage-based payments - payments for the amount of service used, based on some unit of measure One-time payments - Usually fees for setup but also for temp...

1. Product Improvement Changing product features or content Increase stickiness Increase cost for a customer to switch to an alternative (providing valuable features that are hard to reproduce or difficult to transfer from one system to another) 2....

Published: 2007 Abstract Recommendation Systems ถูกใช้งานในการทำ Personalized Services เยอะขึ้นเรื่อยๆ เทคนิคที่มักจะใช้บ่อยคือ Collaborative filtering และ naive Bayesian classification ซึ่งสองเทคนิคนี้ก็มีปัญหาหลายอย่าง เช่น cold-start, หรือ เรื่อง...

Predicting churn doesn't work (well) Predicting churn should be targeted with the actionable insight, not just binary results (churn or not) Churn is hard to predict with high accuracy customers may took long time to cancel Utility...

What is XYZ Analysis XYZ Analysis is an inventory categorisation technique based on the demand variations overtime, the demand variations may caused by different factors such as seasonality, competitor action, or any events happen in a specific period. ...

We can use Matplotlib Dates API together with major and minor tick formatter. Preparing data import pandas as pd import numpy as np import seaborn as sns random_value = np.random.rand(365, 1) df = pd.DataFrame(random_value, columns=['Value']) ...

We can use ABC Analysis to classify the most important product, but we may need to take time into account where XYZ Analysis come in place. ABC-XYZ Principle Using both of ABC Analysis and XYZ Analysis together will give us a big picture on which product ...

Authors of Book - An Elegant Puzzle

Authors of Book - The Truth About Employee Engagement

Summary Author: Patrick Lencioni My review: ⭐⭐⭐ (3/5) Summary Anonumity (การไม่มีตัวตน) Knowing other people as a person We do not need to be different person at work Irrelevance (การไม่ส่งผลกระทบ) We often make a d...

Summary Author: Will Larson My review: Summary In Progress … Notes 3.14 Time Management 1. Quarterly Time Retrospective This idea is interesting to try, it should help reflect how I spend my time. I'll share again the result after applying it. ...

Recently I came across the use-case that, we would like to see the page "BEFORE" the users made the purchase event from the interactions data. To do this, one of the option is "Window function". Window Function There are really useful...

Parametric Associate with normal distributions (mean, median, and mode) Non-Parametric Cannot assume the data comes from a normal distribution For example, a study comparing ranked personal preferences will want to use non-parametric statistics to...

Currently Reading Fearless Organization Book - An Elegant Puzzle Read Book - Zapp - The lightning of empowerment Book - The Truth About Employee Engagement The Manager's Path - Camile Fournier The Making of a Manager - Julie Zhuo I'...

What is ABC Analysis ABC Analysis is an inventory categorisation technique based on the Preto-rules (20%/80%) ABC Analysis divides an inventory into three categories “A items” – 20% of the items accounts for 70% of the annual consumption val...

Netflix มี ranking algorithm หลากหลายตัว และแต่ละตัวถูกสร้างมาด้วยจุดประสงค์ที่ต่างกัน Top Picks - เรียงตามความชอบส่วนบุคคล Trending Now - ใช้ความ popular เข้ามาคิดด้วย เพื่อที่จะวัดผลว่า ranking algorithm แบบไหนสำเร็จ ก็ต้อง...

Interleaved experiment มักใช้ในการวัด ranking algorithm (e.g., search engines) ใน interleaved experiment เราจะมี ranking algorithm 2 ตัว A และ B Algorithm A โชว์ผล A1, A2, A3, A4 Algorithm B โชว์ผล B1, B2, B3, B4 เราจะเอา Algorithm...

Reference NLP Text Preprocessing: A Practical Guide and Template Text Preprocessing in Python: Steps, Tools, and Examples 11 Techniques of Text Preprocessing Using NLTK in Python

How to connect your apple tv to hotel wifi networks - Clairetreyz

My Personal note, also in Thai :D 1 - No one's crazy No one's crazy ไม่มีใครบ้าหรอก เราตัดสินจากมุมที่เรามี สิ่งแวดล้อมที่เราเติบโต และข้อมูลในขณะนั้น ทำไมคนจนซื้อหวย เพราะว่า มันเป็นการตัดสิน ใจสิ่งแวดล้อมเค้า คนที่เติบ...

This note will be in Thai :D 📖 ช่วงอาทิตย์ที่ผ่านมาได้ไปอ่านหนังสือเล่มนึงชื่อ obesity code ของ Jason Fung 🤔 หนังสือพาเราไปทำความเข้าใจ สาเหตุ ของโรคอ้วน โดยเอาหลักฐานการวิจัยมาให้ดู ⚠️ รายละเอียดในหนังสือเยอะ แนะนำให้ไปอ่านเองดีกว่า ตัวอย่างเช่น การ...

Resources Woobull An introduction to on-chain analysis - Sukanya Singh MLQ.ai - What is Crypto on-chain analysis LearnCrypto.com - What is on-chain analysis Data and Analysis Into The Block Xangle CryptoQuant Dune Analytics Flipside C...

What is imputation How to impute values Mean/Median/Mode Hot-Deck Model-based Proper multiple Stochasic Regression Pattern Submodel Approach 6 Different Ways to Compensate for Missing Values In a Dat...

Why MLOps Drivers for MLOps Data Scientist spend too much time on deployment >50% of their time Deployment Gap Evolution of MLOps

What is Machine Learning Operations Why MLOps End-to-end Machine Learning Lifecycle MLOps Principles Automation Further reading Continuous X Continuous Integration (CI) Continuous Delivery (CD) Continuous Training (CT) Continuous Monito...

Model Engineering Model training Include feature engineering, and the hyperparameter tuning https://netflixtechblog.com/machine-learning-platform-meetup-ddec090f3c17 Details Feature Engineering ...

Model Deployment Model Serving Model Performance Monitoring https://engineering.linkedin.com/blog/2021/model-health-assurance-at-linkedin Model Performance Logging

Data Engineering Data Ingestion Includ synthetic data generations, or data enrichment Details Data Sources Identifications Space Estimation Space Location Obtaining Data Backup d...

End-to-end lifecycle MLOps - Data Engineering MLOps - Model Engineering MLOps - Model Deployment

What is Machine Learning Operations A set of culture and practice that aims at unifying ML system development and operations, automate and monitoring all steps including integration, testing, releasing, deployment, and infrastructure management. Wikipedia...

Resources CS 329S: Machine Learning Systems Design AWS re:Invent 2020: Designing better ML systems: Learning from Netflix Standford MLSys Seminar: Chip Huyen Principles of Good Machine Learning Systems Design 6 Little-Known Challenges After Deplo...

Data Versioning Version control is an important component when working with Machine Learning. It also increase the speed of development and reduce mistakes Increase the speed of development Reproduciblity Data Sharing Change Comp...

Resources Top Five Lessons from Running A/B Test on the World's Largest Professional Network

Resources How to Test Machine Learning Code and Systems Testing ML - Repostory Effective testing for machine learning systems Beyond Accuracy: Behavioral Testing of NLP Models with CheckList Temporal Leakage Feature Leakage

A resources for NFT NFT A beginner's guide to NFTs by Linda Xie Explain It Like I Am 5: NFTs NFT Music Music NFT Landscape by Coopahtroopa Audius - A Decentralized Protocol for Audio Content - White Paper How do you Earn from Aud...

What is Embedding Google ML Crash Course - Embedding said An embedding is a relatively low-dimensional space into which you can translate high-demensional vectors. Embeddings make it easier to do machine learning on large inputs like sparse vectors rep...

Typically Batch Pre-computed Decouple compute from serving Lower operational load Make prediction for everyone Real-time Responsive to time-sensitive contex...

Reference J.-T. Huang et al., “Embedding-based Retrieval in Facebook Search,” Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2553–2561, Aug. 2020, doi: 10.1145/3394486.3403305. #search #rec...

Retrieval Ranker Reference P. Covington, J. Adams, and E. Sargin, “Deep Neural Networks for YouTube Recommendations,” in Proceedings of the 10th ACM Conference on Recommender Systems, Boston Massachusetts USA, Sep. 2016, pp. 191–198. doi: 10.1145/...

Reference J. Wang, P. Huang, H. Zhao, Z. Zhang, B. Zhao, and D. L. Lee, “Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba,” arXiv:1803.02349 [cs], May 2018, Accessed: Jan. 25, 2022. [Online]. Available: http://arxiv.org/abs/1803...

Note Reference Q. Chen, H. Zhao, W. Li, P. Huang, and W. Ou, “Behavior Sequence Transformer for E-commerce Recommendation in Alibaba,” arXiv:1905.06874 [cs], May 2019, Accessed: Nov. 06, 2021. [Online]. Available: http://arxiv.org/abs/1905.06874 #r...

Note Reference Y. Feng, B. Hu, F. Lv, Q. Liu, Z. Zhang, and W. Ou, “ATBRG: Adaptive Target-Behavior Relational Graph Network for Effective Recommendation,” Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Info...

https://doordash.engineering/2020/12/15/understanding-search-intent-with-better-recall/ Embedding-based Retrieval in Facebook Search ElasticSearch tips S. Lamkhede and C. Kofler, “Recommendations and Results Organization in Netflix Search,” arXiv:2105...

System Design for Recommendations and Search - Eugene Yan System Design for Recommendations and Search What's trending on Netflix System Architectures for Personalization and Recommendation - Netflix Tech Blog #recommender-system #recommender-sy...

Learning a Personalized Homepage - Netflix Tech Blog Recent Trends in Personalization: A Netflix Perspective Personalization at Netflix - Making Stories Travel Personalized Page Generation for Browsing Recommendations Homepage Personalization at Spo...

Accuracy is not enough Introduce intra-list similarity metric to measure diversity which improves user satisfaction Propose topic diversification method to increase diversity The Intra-List Similarity Metric and the process of Topic Diversification...

Experimental design Goal: Measure the effectiveness of the signup process Metrics: User effort - How hard was it to sign up Accuracy - How well can the system make recommendations to the users ...

Conclusion We need to judge the quality of recommendations as users see them: as recommendation lists We need to create a variety of metrics that act on recommendation lists (e.g., intra-list similarity as introduced in Improving recommendatio...

Reference G. Tolomei, M. Lalmas, A. Farahat, and A. Haines, “You Must Have Clicked on this Ad by Mistake! Data-Driven Identification of Accidental Clicks on Mobile Ads with Applications to Advertiser Cost Discounting and Click-Through Rate Prediction,” a...

Summary SRM is a mismatch between the expected sample ratio and the observed sample ratio SRMs are a common data quality issues Approximately 6% of experiments at Microsoft exhibit an SRM SRMs cause a selection bias that invalidates any c...

Recommender System - Accuracy or Relevance The recommendation suggested to a user should be relevant because that a user has a high propensity to purchase the recommended items and rate them highly. Prediction accuracy is not enough ...

Supports Beyond accuracy - evaluating recommender systems by coverage and serendipity Serendipity - Unexpected and usefulness (usefulness judged by user) Being accurate is not enough - how accuracy metrics have hurt recommender syst...

Supports Beyond accuracy - evaluating recommender systems by coverage and serendipity Coverage Prediction Coverage Catalogue Coverage Effective Catalog Size #recommender-system #recomme...

Supports Being accurate is not enough - how accuracy metrics have hurt recommender systems Accuracy hurt recommender system #recommender-system #recommender-system/quality #recommender-system/accuracy

Google News Personalization scalable online collaborative filtering Personalized News Recommendation Based on Click Behavior - Google Offline and online evaluation of news recommender systems at swissinfo A Live Comparison of Methods for Personalized ...

Optimization metrics quantify how users engage within a session and act as proxy of satisfaction save long click post follow click to stream impression to click dwell time percentage completion ...

** Reference R. Mehrotra, M. Lalmas, D. Kenney, T. Lim-Meng, and G. Hashemian, “Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations,” in The World Wide Web Conference on  - WWW ’19, San Francisco, CA,...

Web-usage mining Focus on efficiency, effectiveness, actionable marketing intelligence Efficiency Measured by customer satisfaction usability Mining by using click-stream sequence analysis Effe...

Promotions and advertisements Search Recommender Systems Pricing and assortment Reference Algorithmic Marketing book - ILYA KATSOV

Netflix Ads Personalization Solution - 2017 Ads Personalization at Spotify - 10/23 Real time ads personalization at Spotfy You Must Have Clicked on this Ad by Mistake Related Adobe Personalization 2020 Survey of Consumers and Marketers Realtime...

Goals: Recent and Fresh, Diverse and Relevant 2 Phase Candidate generations Co-visitation Recommendation (association rule mining) Ranking Variety of signals for relevance and diversit...

Personalize Video Ranker: PVR Orders the entire catalog of videos (or subsets selected by genre or another filtering) Top-N Video Ranker Find the best few personalized recommendations in the entire catalog for each member ...

Summary Tensorflow Serving can poll updated model automatically Tensorflow Serving can poll from Cloud Storage (e.g. AWS, GCP) Tensorflow Serving can deploy multiple models, or multiple model’s version Tensorflow Serving support batch inf...

TF-IDuF achieved 5.14% CTR, equal to TF-IDF (5.09% CTR) Reference J. Beel, S. Langer, and B. Gipp, “TF-IDuF: A Novel Term-Weighting Scheme for User Modeling based on Users’ Personal Document Collections,” p. 8, 2017. #recommender-system

Generative Clustering Model Objective To maximize the total intra-cluster coherence Use-cases Naive Bayes for ranking (Similar product) Inferring latent product for unseen one ...

Offline process, generates long-term cluster based on product’s title e.g. c1 = {nike, air-max, white, gray, running} Nike white gray 7 mesh Air Max -> c1 WOMENS 10.5 Nike Air Max...

Co-purchase mining approach Results ebay.com improve 30% in CTR ebay.de improve 42% in CTR Reference J. Katukuri, T. Konik, R. Mukherjee, and S. Kolay, “Post-purchase recommendations in large-scale online marketplaces,” in 2015 IEEE Internation...

Bayesian framework The log analysis reveals that the click distributions of individual users are influenced by the local news trend Decompose users news interests User’s genuine interests (Long-term) ...

Recall + Ranking Recall: Candidate Generation Coviewed items A pair of items that have been frequently viewed together in the same browsing session by multiple users Filter out covered f...

Context-tree recommender systems which profile the users in real-time the CTR overestimates the actual impact for popular items and thus gives a skewed impression of the actual performance. Therefore, only comparing the actual article views can accurately...

6 measures of similarity for recommendations in social network L1-Norm L2-Norm Pointwise Mutual-Information: Positive correlations Pointwise Mutual-Information: positive and negative correlations Salton (IDF) Log-Odd...

Customer who viewed this also viewed Related work The Youtube video recommendation system 2010 There are other interesting work which is Fake Co-visitation Injection Attacks to Recommendr systems #recommender-system/videos #youtube #recommend...

1. Coverage The coverage of a recommender is a measure of the domain of items over which the system can make recommendations 2 concepts The percentage of the items for which the system is able to generate the recommendation (referred as &quo...

Adoption and Conversion Rates often require a domain- and application-specific definition. Supports The Youtube video recommendation system 2010 Long CTR - only counting clicks that led to watches of a substantial fraction of the video ...

Offline evaluation contradict with online evaluation Both CTR and MAP never contradicted each other It could still be possible that MAP over users will differ Research Questions Why do offline evaluations only sometimes accurately ...

Note: Hybrid of collaborative-filtering and a content-based method that leverage Wikipedia-based concept features Post-processed by a novel Bayesian remapping technique Baseline Popular Result Increase in CTR of 37% (Hybrid with Wiki vs. popular) ...

Evaluate recommender system offline evaluation online evaluation user studies Result from offline sometimes contradict from online and user studies Human factors imperfection of offline datasets Of...

Subscription business models are based on the idea of selling a product or service to receive monthly or yearly recurring subscription revenue. They focus on customer retention over customer acquisition. Metrics Monthly Recurring Revenue (MRR) Annual...

Retention rate is the percentage of customers a business retains over a given period. This is a critical success metric for a subscription-based businesses model. Logically, a high retention rate would show that a business has a low churn rate. Retention R...

What is being measured Click Through Rate Easy to measure and established Not the ultimate goal Adoption and Conversion Rates Easy to measure Requires a domain-specific definition Requires interpretation...

Problems Scalability Item Churn Baseline Decayed popularity Result Increase in CTR by 38% compared between the proposed method and baseline popularity. Reference A. S. Das, M. Datar, A. Garg, and S. Rajaram, “Google news personalization: scala...

A metric that describes how spread viewing is across the items in our catalog. If most viewing comes from a single item, it will be close to 1. If all items generate the same amount of viewing, it is close to the number of videos in the catalog. p is the...

Click-Through-Rate (CTR) is the number of clicks that the item receives divided by the number of times the item is shown. clicks / impressions = CTR If we have 5 clicks and 100 impressions, our CTR would be 5%. Supports Google News Personalization sc...

Summary Feature crossing is one of the feature engineering technique. Feature crossing can capture the interaction between two or more categorical features. Feature crossing helps model train faster, easier to predict target label. Featur...