Berton Shares Beyer Lecture Plot on SSL Limits

This is more an example of a bad filter rather than why you shouldn't filter. That said, I absolutely agree with @giffmana that overfiltering is commonn and that lots of highly useful data is filtered out in most publicly curated datasets. There other public examples of this:… Show more

This example also highlights the perils of poor filtering, which can easily introduce unintended bias into models, and make them perform disparately for different groups of people. This is why curation to focus on non-English capabilities and populations is so critical if we… Show more

+1 to framing problem the data selection as opposed to data filtering - this is a very important distinction I think the whole reason generic data filters have often failed with more scale is that in the good-at-everything model setting, when you have the compute to train on… Show more

Of course there's a lot more to learn in there, but these are the points I found most interesting as someone who's been in vision for a while @giffmana thanks for the great presentation! Hopefully this summary properly reflects your views… Show more

@arimorcos @giffmana Agree a ton on the selection and sources part:) Our DCVLM’s findings resonate most of what you suggested: https://twitter.com/vishaal_urao/status/2072749394387149125

@sjoshi804 @arimorcos And data selection is an infinite problem because it’s conditional on an ever moving goal!