Vokenization: Chilankhulo chomwe AI angachiwone

• Author:
• Dzina la wolemba

  Quantumrun Foresight
• Mwina 9, 2023

Natural language processing (NLP) yathandiza ma artificial intelligence (AI) kuti aphunzire zolankhula za anthu pomvetsetsa mawu ndikufananiza nkhani ndi malingaliro. Choyipa chokha ndichakuti machitidwe a NLP awa amangotengera zolemba. Vokenization yatsala pang'ono kusintha zonsezi.

Vokenization context

Mapulogalamu awiri a makina ophunzirira makina (ML) nthawi zambiri amagwiritsidwa ntchito pophunzitsa AI kukonza ndi kumvetsetsa chilankhulo cha anthu: OpenAI's Generative Pre-trained Transformer 3 (GPT-3) ndi Google's BERT (Bidirectional Encoder Representations from Transformers). Mu terminology ya AI, mawu omwe amagwiritsidwa ntchito pophunzitsa NLP amatchedwa zizindikiro. Ofufuza ochokera ku yunivesite ya North Carolina (UNC) adawona kuti mapulogalamu ophunzirira malemba ndi ochepa chifukwa sangathe "kuwona," kutanthauza kuti sangathe kujambula zithunzi ndi mauthenga. 

Mwachitsanzo, ngati wina afunsa GPT-3 kuti mtundu wa nkhosa ndi chiyani, dongosololi nthawi zambiri limayankha "wakuda" ngakhale litakhala loyera. Yankho ili chifukwa chakuti dongosolo lolemba malemba lidzagwirizanitsa ndi mawu akuti "nkhosa zakuda" m'malo mozindikira mtundu wolondola. Mwa kuphatikiza zowoneka ndi ma tokeni (voken), machitidwe a AI amatha kumvetsetsa mawu. Vokenization imagwirizanitsa ma voken mu machitidwe odziyang'anira a NLP, kuwalola kuti apange "nzeru."

Kuphatikiza mitundu ya zilankhulo ndi masomphenya apakompyuta si lingaliro latsopano, ndipo ndi gawo lomwe likukula mwachangu mu kafukufuku wa AI. Kuphatikiza kwa mitundu iwiriyi ya AI kumawonjezera mphamvu zawo. Mitundu ya zilankhulo monga GPT-3 imaphunzitsidwa kudzera m'maphunziro osayang'aniridwa, zomwe zimawalola kuti azikula mosavuta. Mosiyana ndi izi, zitsanzo za zithunzi monga machitidwe ozindikiritsa zinthu amatha kuphunzira kuchokera ku zenizeni ndipo sadalira zomwe zimaperekedwa ndi malembawo. Mwachitsanzo, zitsanzo za zithunzi zimatha kuzindikira kuti nkhosa ndi yoyera poyang’ana chithunzi.

Zosokoneza

Njira ya vokenization ndiyabwino kwambiri. Ma Vokens amapangidwa pogawira zithunzi zofananira kapena zoyenera kuzizindikiro zachilankhulo. Kenako, ma aligorivimu (vokenizer) amapangidwa kuti apange ma voken kudzera mumaphunziro osayang'aniridwa (palibe magawo / malamulo omveka bwino). Nzeru AI yophunzitsidwa kudzera mu vokenization imatha kuyankhulana ndikuthetsa mavuto bwino chifukwa amamvetsetsa mozama nkhani. Njirayi ndi yapadera chifukwa sikuti imangoneneratu zizindikiro za chinenero komanso imaneneratu zizindikiro zazithunzi, zomwe ndi zomwe zitsanzo zachikhalidwe za BERT sizingathe kuchita.

Mwachitsanzo, othandizira ma robot azitha kuzindikira zithunzi ndikuyendetsa njira bwino chifukwa amatha "kuwona" zomwe zimafunikira kwa iwo. Machitidwe anzeru zopangira ophunzitsidwa kulemba zomwe ali nazo azitha kupanga zolemba zomwe zimamveka ngati zamunthu, zokhala ndi malingaliro oyenda bwino, m'malo mwa ziganizo zosagwirizana. Poganizira kuchuluka kwa mapulogalamu a NLP, kuvomera kungayambitse ma chatbots ochita bwino, othandizira enieni, kuzindikira zachipatala pa intaneti, omasulira a digito, ndi zina zambiri.

Kuphatikiza apo, kuphatikiza kwa masomphenya ndi kuphunzira chilankhulo kukuchulukirachulukira pakugwiritsa ntchito kujambula kwachipatala, makamaka pakuzindikira zithunzi zachipatala. Mwachitsanzo, ofufuza ena akuyesa njira iyi pazithunzi za radiograph ndi mafotokozedwe a malemba, kumene magawo a semantic amatha nthawi yambiri. Njira ya vokenization imatha kupititsa patsogolo mawonekedwe awa ndikuwongolera kujambula kwachipatala pogwiritsa ntchito zidziwitso.

Mapulogalamu a vokenization

Ntchito zina zoyitanitsa zisankho zitha kukhala:

• Ma chatbots anzeru omwe amatha kukonza zowonera, zithunzi, ndi zomwe zili patsamba. Makasitomala othandizira ma chatbots, makamaka, amatha kupangira molondola malonda ndi ntchito.
• Omasulira a digito omwe amatha kukonza zithunzi ndi makanema ndikupereka zomasulira zolondola zomwe zimatengera chikhalidwe ndi zochitika.
• Ma social media bot scanner amatha kusanthula malingaliro onse pophatikiza zithunzi, mawu ofotokozera, ndi ndemanga. Pulogalamuyi ikhoza kukhala yothandiza pakuwongolera zinthu zomwe zimafunikira kuwunika kwazithunzi zoyipa.
• Kuchulukitsa mwayi wantchito wa masomphenya apakompyuta ndi akatswiri ophunzirira makina a NLP ndi asayansi a data.
• Oyambitsa omwe amamanga pamakina a AI awa kuti awagulitse kapena kupereka mayankho makonda pamabizinesi.

Mafunso oti muyankhepo

• Kodi mukuganiza kuti vokenization ingasinthe bwanji momwe timalumikizirana ndi ma robot?
• Kodi kutulutsa mawu kungasinthe bwanji momwe timachitira bizinesi ndikulumikizana ndi zida zathu (mafoni a m'manja ndi zida zanzeru)?